From dd442d22ffb179437451676553d15d174ebad3f6 Mon Sep 17 00:00:00 2001 From: sameeeeeeep <191771413+sameeeeeeep@users.noreply.github.com> Date: Tue, 29 Sep 2026 14:22:38 +0530 Subject: [PATCH 1/4] System 1, screen journal, live cards and the Loose Notch - switchboard_decide: typed decisions (noul/choice/score) through a System One model. Hosted TypeSafe Jev via OpenRouter (key checked before any data is sent), or any /v1/systemone server via SYSTEM1_URL. /system1 skill documents when to use it. packages/system1 holds the key saver and an opt-in local engine. - Screen journal (ScreenJournal.swift): opt-in via ~/.relay/journal-on, reads the focused window's text through Accessibility, OCR fallback when that is thin (Chrome), block list, secret masking, 30-day retention, files 0600. - switchboard_recall / switchboard_journal: recall screen history by meaning (local prefilter, then System 1 per line, redacted) and control recording. - Screen watch (watch.mjs, LaunchAgent via install-watch.mjs): one System 1 call per settled window; cards only when a real person asks something of the user (v2 gate), memory.md and board tasks in the vault, answers logged as labels. watch-replay.mjs replays journal windows plus probes to tune triggers. - Notch: lamp eyes inside the dot field while journaling, STOP WATCHING on hover. - Loose Notch phase 1 (LooseNotch.swift): pull the notch down into a free pill, snap to eight anchors, dock back; the notch hides while it is loose. - docs/AUDIO-JOURNAL-SPEC.md: spec for an always-on audio layer (not built). Co-Authored-By: Claude Opus 5.5 --- docs/AUDIO-JOURNAL-SPEC.md | 113 +++++++ packages/menubar/LooseNotch.swift | 195 ++++++++++++ packages/menubar/RelayMenuBar.swift | 66 +++- packages/menubar/ScreenJournal.swift | 304 +++++++++++++++++++ packages/menubar/build.sh | 2 +- packages/switchboard-mcp/install-watch.mjs | 41 +++ packages/switchboard-mcp/journal.mjs | 114 +++++++ packages/switchboard-mcp/package.json | 2 +- packages/switchboard-mcp/switchboard-mcp.mjs | 2 + packages/switchboard-mcp/system1.mjs | 140 +++++++++ packages/switchboard-mcp/watch-replay.mjs | 64 ++++ packages/switchboard-mcp/watch.mjs | 202 ++++++++++++ packages/switchboard-mcp/watch.test.mjs | 81 +++++ packages/system1/install.sh | 16 + packages/system1/serve.py | 84 +++++ packages/system1/set-key.py | 41 +++ plugin/skills/system1/SKILL.md | 64 ++++ 17 files changed, 1526 insertions(+), 5 deletions(-) create mode 100644 docs/AUDIO-JOURNAL-SPEC.md create mode 100644 packages/menubar/LooseNotch.swift create mode 100644 packages/menubar/ScreenJournal.swift create mode 100644 packages/switchboard-mcp/install-watch.mjs create mode 100644 packages/switchboard-mcp/journal.mjs create mode 100644 packages/switchboard-mcp/system1.mjs create mode 100644 packages/switchboard-mcp/watch-replay.mjs create mode 100644 packages/switchboard-mcp/watch.mjs create mode 100644 packages/switchboard-mcp/watch.test.mjs create mode 100644 packages/system1/install.sh create mode 100644 packages/system1/serve.py create mode 100644 packages/system1/set-key.py create mode 100644 plugin/skills/system1/SKILL.md diff --git a/docs/AUDIO-JOURNAL-SPEC.md b/docs/AUDIO-JOURNAL-SPEC.md new file mode 100644 index 0000000..7b6ec2c --- /dev/null +++ b/docs/AUDIO-JOURNAL-SPEC.md @@ -0,0 +1,113 @@ +# Audio Journal: spec (draft, 2026-09-29) + +**Summary.** Add an opt-in "ears" layer next to the screen journal. The mic feeds a tiny voice-activity detector (VAD). Only speech gets transcribed on the Mac by whisper.cpp, then labelled by speaker. The result is written as plain text into the **same** `~/.relay/journal/YYYY-MM-DD.jsonl`, so `switchboard_recall`, `memory.md` and live cards work without changes. No audio is ever saved. The layer is off by default. The default mode is "meetings only", the notch shows it is listening, and other people are told they are being transcribed. Recommended stack for an 8 GB M1: **sherpa-onnx** (Silero VAD + pyannote-segmentation-3.0 + a 3D-Speaker/CAM++ embedding) for VAD and speakers, plus the whisper.cpp + `ggml-base.en.bin` already on the machine for text. All resource numbers below are **estimates** until the Phase 0 benchmark runs. + +## 1. Pipeline + +``` +mic (16 kHz mono, in-memory ring buffer ~30 s) + → VAD (Silero, always on while armed; ~32 ms frames) + → speech segment (cut at pauses; max 30 s) + → whisper.cpp transcribe (base.en + dictionary prompt) ┐ run on the same segment + → speaker embedding → match "me" / named / "Speaker N" ┘ + → journal entry {source:"audio", ...} → buffer freed +``` + +**Capture.** A new `AudioJournal.swift` next to `ScreenJournal.swift` reuses the device selection and the 16 kHz mono PCM format in `DictationAudio.swift` (`DictationMicrophone`, `DictationRecorder`). Audio lives only in a RAM ring buffer. Silence is dropped frame by frame and never leaves the buffer. + +**VAD.** Silero VAD runs on every frame while the layer is armed. It is the only part that is always on. whisper.cpp already includes Silero VAD (`--vad -vm ggml-silero-*.bin`; the installed `whisper-cli --help` lists these flags), but that path only works on files. Streaming needs VAD inside our process, so we use sherpa-onnx's Silero VAD through its Swift API. + +**Transcription.** Each speech segment goes to whisper.cpp with the existing dictionary prompt (`readDictionaryPrompt`, same as `stt.ts`). Spawning `whisper-cli` for every segment reloads the ~148 MB model each time. Instead, keep one warm whisper instance (whisper.cpp's server via the existing `RELAY_LOCAL_STT_URL` path, or linked libwhisper) that runs **only while a session is active** and shuts down after 2 min of silence. If the CLI fallback is used, its temp WAV goes in a 0700 temp dir and is deleted as soon as the call returns. That file is the only moment audio touches disk, and it counts as "transient", not "stored". + +**Diarization and identification. Options:** + +| Option | Who spoke? | Fit for 8 GB M1 | +|---|---|---| +| **sherpa-onnx**: pyannote-seg-3.0 (ONNX) + 3D-Speaker / CAM++ embedding + clustering | Speaker turns **and** stable voice identity (embeddings), so "me" and named people can be matched | ~45 MB of models, C++ runtime, Swift API, no Python, not gated. **Recommended** | +| ECAPA embeddings alone (e.g. SpeechBrain) | Identity only, no overlap handling | Needs PyTorch/Python; the same idea is available inside sherpa-onnx without that stack | +| pyannote 3.1 | Best-known quality | Python + PyTorch, gated on Hugging Face (accept terms + token). Too heavy to keep resident next to everything else on 8 GB | +| whisper.cpp tinydiarize (`-tdrz`) | Marks speaker **turns** only (`[SPEAKER_TURN]`), no identity | Needs the `small.en-tdrz` model (bigger than base.en); described upstream as an experimental prototype. Could not answer "what did Priya say" | + +**Why sherpa-onnx:** one small native library covers VAD, segmentation and embeddings. It adds no Python and no model gating. Identity comes from embeddings, which is what enrolment and naming need. + +**Live vs. second pass.** *Live:* one embedding per speech segment (segments of 1.5 s or longer), matched by cosine similarity against enrolled voices and this session's clusters. Unknown voices become "Speaker 2", "Speaker 3" and so on. *Second pass (plugged in only):* when a session ends, run full pyannote-seg + clustering over the session's segments to fix overlaps and relabel. This means the session's segment audio stays in RAM until the pass finishes (capped, e.g. 60 min; longer sessions skip the pass). This is an open question. + +**"Me" enrolment.** In a one-time notch card, the user reads ~20 s of text aloud. We store one averaged embedding (a list of numbers, not audio) in `~/.relay/voices.json` (0600). **Naming others:** after a session, a card asks "Speaker 2 said '…first line…'. Who is this?" with options *Name them / Skip / Don't remember this voice*. A name is stored as an embedding plus a label, and past lines are relabelled. A voice embedding counts as biometric-like data: it is local only, can be deleted, and is never synced. + +**Journal entry: same file, same shape, plus fields:** + +```json +{"t":"…","app":"Audio","bundle":"switchboard.audio","window":"Zoom call · 14:02","url":null, + "source":"audio","session":"a-7f3c", + "lines":["Me: let's ship Friday","Priya: I'll send the deck tonight"], + "segs":[{"spk":"me","t":"…","text":"let's ship Friday"},{"spk":"v_12","t":"…","text":"I'll send the deck tonight"}]} +``` + +`lines` carries the speaker name as a prefix. Because of that, `journal.mjs` prefilter/recall and `watch.mjs` (which settles on `bundle|window`, reads `lines`, and writes `memory.md`) work **unchanged**. `segs` holds stable speaker ids for per-speaker delete and rename. Only one watch change is needed: it currently follows `journal-on`, and it must also run when `audio-on` exists. Screen-journal rules also apply to audio lines: secret masking on write, and redaction before anything reaches System 1. + +## 2. Resource budget (all estimates, measure in Phase 0) + +| State | CPU (M1) | RAM | Battery | +|---|---|---|---| +| Armed, silence (VAD only) | ~1–3 % of one core (est.) | ~30–60 MB (est.) | small, roughly like a voice-memo app idling (est.) | +| Speech, live transcribe + embed | bursty; whisper base.en on Metal/Neon ~0.1–0.3× real-time (est.) | +250–400 MB while the whisper instance is warm (est.; model file is 148 MB, measured) | a 1 h meeting maybe a few % extra (est.) | +| Second pass (plugged in only) | a short burst per session (est.) | +100–200 MB (est.) | n/a | + +Guard rails: whisper loads only during a session and unloads after 2 min of silence. If memory pressure is warning or worse, pause transcription and keep VAD, and show the state on the notch. **Plugged-in only** is a toggle. **Meeting-only** (the default) arms capture only while a call app is using the mic (Zoom, Meet in a browser, Teams, FaceTime, Slack huddle), detected through Core Audio's "device is running somewhere" signal plus the frontmost app. Phase 0 must check the house rule (never noticeably slow the device) by running the real pipeline during a real call on this Air: Activity Monitor CPU and memory, and `powermetrics` energy. + +## 3. Consent and privacy (first-class) + +- **Off by default.** Turning it on happens through a notch card that explains the behaviour in one line and asks the user to pick a mode. Flag `~/.relay/audio-on`, separate from `journal-on`. +- **Modes:** *Meetings only* (default) · *This session* (on until I stop it, or for N hours) · *Places* (on only on chosen Wi-Fi networks, e.g. office; never at home unless chosen) · *Always* (needs a second confirmation card). +- **Visible indicator.** macOS shows the orange mic dot whenever the mic is in use. We cannot and must not hide it. On the notch, the lime eyes change to show "listening". Proposed: the eyes gain a small pulse under them while armed, which becomes a waveform while speech is being transcribed. Tapping it pauses audio. Screen and audio pause separately, with a long-press to pause both. +- **Never store raw audio.** Text only. RAM buffers are freed per segment. The only file is the CLI's transient temp WAV, deleted immediately. +- **Per-speaker delete.** "Forget Priya" removes her voice embedding and deletes or redacts all `segs`/`lines` spoken by her across the journal. +- **Forget the last N minutes.** This reuses `journal.forget(minutes)`, which already works on the shared file. A notch button offers 5 / 15 / 60 minutes. +- **Exclusions.** App block list (no capture while a chosen app has the mic, e.g. a therapy or telehealth app). Place exclusions (Wi-Fi). Time windows. A "private" keyword: saying "off the record" pauses for 10 min, and this is shown on the notch. +- **Retention:** the same `retentionDays` as the journal (30). A separate shorter value for audio is an option. + +### Recording other people + +Recording or transcribing a conversation without the other people's consent is legally restricted in many jurisdictions. Some require consent from everyone present, and rules differ for calls and in-person talk. This spec does not state what any specific law requires. **Verify locally before shipping.** Product defaults: in meetings, the first time a session starts, a card offers to post a one-line notice in the call chat ("I'm using a local transcriber; say if you'd rather I stop"). In-person modes show a reminder to announce it. Any participant's objection maps to a single action: stop, then forget this session. The onboarding card says plainly that consent is the user's responsibility. + +## 4. Controls, states and reversal + +| State | What the user sees | How to reverse | +|---|---|---| +| Off | no ear mark | Turn on from the notch card or `switchboard_journal` | +| Armed (VAD only) | eyes + pulse, orange dot | tap to pause · turn off | +| Transcribing | waveform under the eyes | pause · "forget last 5 min" | +| Paused (user / "off the record") | pulse dimmed, timer | tap to resume, or it auto-resumes when the timer ends | +| No mic permission | card: "Allow microphone in System Settings" with a button to that pane | grant, then it re-arms automatically | +| Mic busy / device gone (e.g. AirPods switch) | small dot + "mic unavailable" | re-arms when the device returns; follows `DictationMicrophone` choice | +| Dictation active | audio journal yields to dictation | resumes after dictation | +| Low memory | "listening paused, low memory" | auto-resume when pressure is normal | +| whisper / model missing | card: install whisper.cpp + base.en (opt-in, shows size) | install or turn off | +| sherpa models missing | transcribes without speakers ("Speaker ?") + install card | install the ~45 MB models | +| On battery (plugged-in-only mode) | "waiting for power" | plug in, or turn the toggle off | + +The MCP `switchboard_journal` tool gains `audio: on|off|mode`, `forgetSpeaker`, and `renameSpeaker`. + +## 5. Phased build + +0. **Benchmark (1 day):** a script feeds a recorded meeting through sherpa VAD + whisper base.en + embeddings on this Air, and records CPU, RAM, energy and real-time factor. Nothing ships. +1. **Meeting-only text, no speakers:** capture, VAD, whisper, journal entries with `source:"audio"`, notch indicator, forget-N-minutes. This is the smallest useful slice: recall works over meetings. +2. **"Me" vs. "others":** enrolment card plus live embedding matching. +3. **Naming:** post-session naming cards, relabel, per-speaker delete. +4. **Second pass + modes:** plugged-in re-diarization, Places and This-session modes, consent notice in the call chat. +5. **Always mode:** only after measurements show it meets the device-lightness rule. + +## 6. Open questions for the founder + +1. Is "Always" in scope at all, or should the product stay at meetings plus explicit sessions (the stronger privacy story)? +2. Should the second pass keep the session's audio in RAM for up to 60 min, or should we accept live-only speaker labels? +3. Should the consent notice in the call chat be automatic, or a suggestion card every time? +4. Should audio have a shorter retention than screen text? +5. Should audio lines ever reach hosted System 1 (redacted), or should they stay local-only by default? +6. Should we include system audio (other side of a call) via ScreenCaptureKit? That shows macOS's purple indicator and doubles the speakers. + +## 7. What existing products do (verified links only) + +- **Granola** transcribes meeting audio from the Mac's mic and system audio with no bot. Its security page says it "doesn't store the audio from meetings" and keeps only the transcript and notes. Transcription uses third-party providers, so it is not fully local. [granola.ai/security](https://www.granola.ai/security) +- **Limitless Pendant**, a wearable, recorded conversations within mic range. Reviews say a consent mode that would detect new voices was planned but never shipped. Meta acquired Limitless (Dec 2025) and new Pendant sales stopped. These are secondary sources: [fast.io review](https://fast.io/resources/limitless-ai-review-2026/), [legendmemory.ai](https://legendmemory.ai/blogs/the-archive/limitless-pendant-discontinued-the-best-alternatives-in-2026) +- **Building blocks:** [sherpa-onnx speaker diarization](https://k2-fsa.github.io/sherpa/onnx/speaker-diarization/index.html) · [whisper.cpp tinydiarize PR #1058](https://github.com/ggml-org/whisper.cpp/pull/1058) · [pyannote 3.1 (gated)](https://huggingface.co/pyannote/speaker-diarization-3.1) · [whisper.cpp VAD](https://github.com/ggml-org/whisper.cpp) · [macOS orange mic indicator](https://support.apple.com/en-gb/guide/mac-help/mchl50f94f8f/mac) diff --git a/packages/menubar/LooseNotch.swift b/packages/menubar/LooseNotch.swift new file mode 100644 index 0000000..56b36e4 --- /dev/null +++ b/packages/menubar/LooseNotch.swift @@ -0,0 +1,195 @@ +// LooseNotch — phase 1 of "The Loose Notch" PRD: the notch's dot field (with its eyes) can be pulled down +// off the notch into a free pill, dragged anywhere, snapped to eight anchors, and docked back. +// +// Shape of the change: the docked notch window (the orb) never moves, because cards, the panel and God's +// drops are all positioned against it. Detaching shows a SEPARATE small panel (the creature) and makes the +// notch window invisible and click-through (founder: "removed, not in both places"); docking reverses both. The creature panel is only as big as the +// pill plus a margin, so it never swallows clicks elsewhere (lesson from the full-screen cat overlay). +// +// pull the notch down past 24 pt → loose · drag → move · drop near an anchor → snap +// drop in the notch zone, double-click, or hover → DOCK → docked · position kept in ~/.relay/presence.json +import AppKit +import SwiftUI + +@MainActor +final class LooseNotch { + static let shared = LooseNotch() + + private var panel: NotchPanel? + private weak var model: Model? + private var notchFrame: () -> NSRect = { .zero } + private var onOpen: () -> Void = {} + private var setNotchGone: (Bool) -> Void = { _ in } // loose = the notch leaves its spot entirely (never in both places) + private var grab = CGPoint.zero // pointer offset inside the pill while dragging it + static let size = CGSize(width: 132, height: 40) // pill 120×32 + 6 pt margin all round + private let pullThreshold: CGFloat = 24 + private let dockRadius: CGFloat = 60 + private let snapRadius: CGFloat = 140 + + private var statePath: String { (NSHomeDirectory() as NSString).appendingPathComponent(".relay/presence.json") } + + func attach(model: Model, notchFrame: @escaping () -> NSRect, onOpen: @escaping () -> Void, + setNotchGone: @escaping (Bool) -> Void) { + self.model = model; self.notchFrame = notchFrame; self.onOpen = onOpen; self.setNotchGone = setNotchGone + if let d = FileManager.default.contents(atPath: statePath), + let j = try? JSONSerialization.jsonObject(with: d) as? [String: Any], j["mode"] as? String == "free", + let x = j["x"] as? Double, let y = j["y"] as? Double { + let origin = CGPoint(x: x, y: y) + if NSScreen.screens.contains(where: { $0.visibleFrame.insetBy(dx: -4, dy: -4).contains(NSRect(origin: origin, size: Self.size)) }) { + show(at: origin) + } + } + } + + // MARK: pulling it off the notch (called by the orb's drag gesture) + + func pullFromNotch(translation: CGSize, ended: Bool) { + let loose = model?.creatureLoose ?? false + if !loose { + guard translation.height > pullThreshold else { return } + grab = CGPoint(x: Self.size.width / 2, y: Self.size.height / 2) + show(at: originUnderPointer()) + } else { + panel?.setFrameOrigin(originUnderPointer()) + } + if ended { settle() } + } + + // MARK: dragging the pill itself + + func dragChanged(first: Bool) { + guard let panel else { return } + let m = NSEvent.mouseLocation + if first { grab = CGPoint(x: m.x - panel.frame.minX, y: m.y - panel.frame.minY) } + panel.setFrameOrigin(originUnderPointer()) + } + func dragEnded() { settle() } + + func open() { onOpen() } + + func dock() { + panel?.orderOut(nil) + model?.creatureLoose = false + setNotchGone(false) + save(mode: "docked", origin: nil) + } + + // MARK: internals + + private func originUnderPointer() -> CGPoint { + let m = NSEvent.mouseLocation + return CGPoint(x: m.x - grab.x, y: m.y - grab.y) + } + + private func show(at origin: CGPoint) { + if panel == nil { + let p = NotchPanel(contentRect: NSRect(origin: origin, size: Self.size), styleMask: [.borderless, .nonactivatingPanel], + backing: .buffered, defer: true) + p.isOpaque = false + p.backgroundColor = .clear + p.hasShadow = false + p.level = .popUpMenu + p.collectionBehavior = [.canJoinAllSpaces, .fullScreenAuxiliary, .transient] + p.acceptsMouseMovedEvents = true + if let model { p.contentView = NoInsetHostingView(rootView: CreatureView(model: model, loose: self)) } + panel = p + } + panel?.setFrameOrigin(origin) + panel?.orderFrontRegardless() + model?.creatureLoose = true + setNotchGone(true) + } + + /// Drop logic: back in the notch zone → dock; near one of eight anchors → snap there; else stay put. + private func settle() { + guard let panel, let screen = panel.screen ?? NSScreen.main else { return } + let f = panel.frame + let notch = notchFrame() + if hypot(f.midX - notch.midX, f.maxY - notch.minY) < dockRadius || f.maxY > screen.visibleFrame.maxY + 4 { + dock(); return + } + let v = screen.visibleFrame.insetBy(dx: 16, dy: 16) + let w = f.width, h = f.height + let anchors: [CGPoint] = [ + CGPoint(x: v.minX, y: v.maxY - h), CGPoint(x: v.midX - w / 2, y: v.maxY - h), CGPoint(x: v.maxX - w, y: v.maxY - h), + CGPoint(x: v.minX, y: v.midY - h / 2), CGPoint(x: v.maxX - w, y: v.midY - h / 2), + CGPoint(x: v.minX, y: v.minY), CGPoint(x: v.midX - w / 2, y: v.minY), CGPoint(x: v.maxX - w, y: v.minY), + ] + var target = CGPoint(x: min(max(f.minX, v.minX), v.maxX - w), y: min(max(f.minY, v.minY), v.maxY - h)) + if let near = anchors.min(by: { hypot($0.x - f.minX, $0.y - f.minY) < hypot($1.x - f.minX, $1.y - f.minY) }), + hypot(near.x - f.minX, near.y - f.minY) < snapRadius { + target = near + } + NSAnimationContext.runAnimationGroup { ctx in + ctx.duration = 0.18 + panel.animator().setFrameOrigin(target) + } + save(mode: "free", origin: target) + } + + private func save(mode: String, origin: CGPoint?) { + var j: [String: Any] = ["mode": mode] + if let o = origin { j["x"] = o.x; j["y"] = o.y } + if let d = try? JSONSerialization.data(withJSONObject: j) { try? d.write(to: URL(fileURLWithPath: statePath)) } + } +} + +/// The loose notch: the same lamp field and eyes as the notch, cut to a pill. Click opens the panel, +/// double-click docks, drag moves it; hover shows STOP WATCHING (when journaling) and DOCK. +struct CreatureView: View { + @ObservedObject var model: Model + let loose: LooseNotch + @State private var hovering = false + @State private var dragging = false + var body: some View { + let tint = model.running ? (model.signedIn ? Color.lime : Color.danger) : Color.inkFaint + let shape = Capsule() + ZStack { + shape.fill(Color.page) + NotchField(accent: tint, working: model.working, animated: model.running, eyes: true) + .padding(.horizontal, 6).padding(.vertical, 3) + .clipShape(shape) + shape.stroke(tint.opacity(hovering ? 0.55 : 0.22), lineWidth: hovering ? 1.1 : 0.75) + if hovering && !dragging { + // The pill is small: icon buttons, with tooltips, instead of the notch's word pills. + HStack(spacing: 8) { + if model.journalOn { + CreatureIconButton(symbol: "eye.slash", help: "Stop watching (pause the screen journal)") { + try? FileManager.default.removeItem(atPath: (NSHomeDirectory() as NSString).appendingPathComponent(".relay/journal-on")) + } + } + CreatureIconButton(symbol: "arrow.up.to.line", help: "Put it back in the notch") { loose.dock() } + } + } + } + .frame(width: 120, height: 32) + .shadow(color: Color.lime.opacity(0.18), radius: 4, y: 1) + .frame(width: LooseNotch.size.width, height: LooseNotch.size.height) + .contentShape(Rectangle()) + .onHover { hovering = $0 } + .gesture(DragGesture(minimumDistance: 3, coordinateSpace: .global) + .onChanged { _ in loose.dragChanged(first: !dragging); dragging = true } + .onEnded { _ in dragging = false; loose.dragEnded() }) + .onTapGesture(count: 2) { loose.dock() } + .onTapGesture { loose.open() } + .help("Drag to move. Double-click to put it back in the notch.") + } +} + +struct CreatureIconButton: View { + let symbol: String + let help: String + var action: () -> Void + @State private var over = false + var body: some View { + Image(systemName: symbol).font(.system(size: 9, weight: .bold)) + .foregroundColor(over ? Color.page : Color.lime) + .frame(width: 22, height: 18) + .background(Capsule().fill(over ? Color.lime : Color.page)) + .overlay(Capsule().stroke(Color.lime.opacity(0.8), lineWidth: 0.8)) + .contentShape(Capsule()) + .onHover { over = $0 } + .onTapGesture { action() } + .help(help) + } +} diff --git a/packages/menubar/RelayMenuBar.swift b/packages/menubar/RelayMenuBar.swift index 8f19bf0..7a1f774 100644 --- a/packages/menubar/RelayMenuBar.swift +++ b/packages/menubar/RelayMenuBar.swift @@ -785,6 +785,8 @@ final class Model: ObservableObject { @Published var selectedVoice: String = "" // the one God speaks in (empty = macOS `say`) @Published var userName: String = "" // what God calls you (~/.relay/profile.json → name) @Published var economy = false // prefer a cheaper/faster model to spend fewer tokens + @Published var journalOn = false // screen journal recording (~/.relay/journal-on) → eyes on the notch + @Published var creatureLoose = false // the Loose Notch pill is off the notch (LooseNotch.swift) → the eyes go with it @Published var regionSelect = false // ⌃⌃ lets you drag a screen region → only that is sent @Published var defaultShare = false // ⌃⌃ auto-shares the whole screen (fn+click then TOGGLES it off) @Published var modelProviders: [ModelProvider] = [] @@ -2388,7 +2390,24 @@ struct NotchField: View { var accent: Color var working: Bool var animated: Bool + /// Screen journal recording → the field's OWN lamps form two eyes (no overlay): lamps inside the eye + /// shapes glow full, the rest of the field dims so they read. Blink + glance ride the same clock. + var eyes: Bool = false @Environment(\.accessibilityReduceMotion) private var reduceMotion + // One eye on the lamp grid: 4 wide × 3 tall with the corners off (an almond). A blink keeps only the middle row. + private static let eyeShape: [(Int, Int)] = [(1, 0), (2, 0), (0, 1), (1, 1), (2, 1), (3, 1), (1, 2), (2, 2)] + private func eyeLamp(_ c: Int, _ r: Int, _ cols: Int, _ rows: Int, _ t: Double) -> Bool { + let blink = t.truncatingRemainder(dividingBy: 4.2) < 0.16 + let glance = [0, 0, -1, 0, 1, 0][Int(t / 5.3) % 6] + let top = max(0, rows / 2 - 1) + let left = cols / 2 - 5 + glance // two 4-wide eyes, 2 lamps apart, centred + for ox in [left, left + 6] { + for (dc, dr) in Self.eyeShape where !(blink && dr != 1) { + if c == ox + dc && r == top + dr { return true } + } + } + return false + } private func bright(_ c: Int, _ r: Int, _ t: Double) -> Double { let cx = Double(c), rx = Double(r) if working { return 0.16 + 0.84 * (0.5 + 0.5 * sin(cx * 0.55 - t * 2.6)) } // travelling wave @@ -2404,7 +2423,8 @@ struct NotchField: View { for c in 0..() + private var lastOn: Bool? + private var lastOCRKey = "" + private var lastOCR = Date.distantPast + private var askedScreenPermission = false + /// Called on the main thread whenever recording turns on/off (the flag file appears/disappears). + var onRecordingChange: ((Bool) -> Void)? + private(set) var running = false + + private var relay: String { (NSHomeDirectory() as NSString).appendingPathComponent(".relay") } + var flagPath: String { (relay as NSString).appendingPathComponent("journal-on") } + private var dir: String { (relay as NSString).appendingPathComponent("journal") } + + static let defaultBlock: Set = [ + // password managers + keychain + "com.1password.1password", "com.agilebits.onepassword7", "com.bitwarden.desktop", "com.apple.keychainaccess", + "com.apple.passwords", "com.dashlane.dashlanephonefinal", "com.lastpass.lastpassmacdesktop", + // private messaging + "com.apple.mobilesms", "net.whatsapp.whatsapp", "desktop.whatsapp", "org.whispersystems.signal-desktop", + "ru.keepcoder.telegram", "com.tdesktop.telegram", + // system security / login UI, and ourselves (our own UI is noise) + "com.apple.systempreferences", "com.apple.securityagent", "com.apple.loginwindow", "com.apple.systemsettings", + "com.relay.menubar", + ] + static let defaultBlockWords = ["private browsing", "incognito", "inprivate", "bank", "netbanking", "paypal", + "password", "sign in", "log in", "otp", "one-time", + "api key", "secret", "credential", "token"] + + // MARK: lifecycle + + /// Arms the (cheap) triggers once at launch. Recording itself follows the flag file live: create + /// ~/.relay/journal-on to start, delete it to stop — no app restart. + func start() { + guard !running else { return } + running = true + NSWorkspace.shared.notificationCenter.addObserver(self, selector: #selector(appActivated), + name: NSWorkspace.didActivateApplicationNotification, object: nil) + let t = DispatchSource.makeTimerSource(queue: queue) + t.schedule(deadline: .now() + 3, repeating: 3) + t.setEventHandler { [weak self] in + guard let self else { return } + let on = FileManager.default.fileExists(atPath: self.flagPath) + if on != self.lastOn { self.lastOn = on; DispatchQueue.main.async { self.onRecordingChange?(on) } } + self.tick(force: false) + } + t.resume() + titleTimer = t + queue.async { self.prune() } + godLog("screen journal armed (records while \(flagPath) exists)") + } + + func stop() { + guard running else { return } + running = false + NSWorkspace.shared.notificationCenter.removeObserver(self) + titleTimer?.cancel(); titleTimer = nil + godLog("screen journal OFF") + } + + @objc private func appActivated(_ note: Notification) { queue.asyncAfter(deadline: .now() + 1) { self.tick(force: true) } } + + // MARK: sampling + + private func tick(force: Bool) { + guard running, FileManager.default.fileExists(atPath: flagPath), AXIsProcessTrusted(), let app = NSWorkspace.shared.frontmostApplication else { return } + let bundle = (app.bundleIdentifier ?? "unknown").lowercased() + let cfg = config() + if cfg.block.contains(bundle) { return } + let axApp = AXUIElementCreateApplication(app.processIdentifier) + AXUIElementSetMessagingTimeout(axApp, 0.25) // an unresponsive app must not stall the journal + // Electron apps (Slack, Claude, Notion, VS Code…) hide their text from AX until asked. This flag is + // Electron's own opt-in; we avoid AXEnhancedUserInterface, which breaks window animations. + if electronAsked.insert(app.processIdentifier).inserted { + AXUIElementSetAttributeValue(axApp, "AXManualAccessibility" as CFString, kCFBooleanTrue) + } + guard let win = jElement(axApp, kAXFocusedWindowAttribute as String) else { note("skip \(bundle): no focused window"); return } + let title = jString(win, kAXTitleAttribute as String) ?? "" + let key = bundle + "|" + title + let stale = Date().timeIntervalSince(lastRead) > 30 + guard force || key != lastKey || stale else { return } + lastKey = key; lastRead = Date() + + let url = jURL(win) + let haystack = (title + " " + (url ?? "")).lowercased() + if let w = cfg.blockWords.first(where: { haystack.contains($0) }) { note("skip \(bundle): block word '\(w)'"); return } + + let started = Date() + var lines = readText(win) + var via = "ax" + // Thin AX text (Chrome and other apps that hide page text from accessibility) → read the pixels: + // on-device Vision OCR of just this window. On a window change, and at most once a minute otherwise. + if lines.count < 8 || lines.reduce(0, { $0 + $1.count }) < 300, + key != lastOCRKey || Date().timeIntervalSince(lastOCR) > 60 { + lastOCRKey = key; lastOCR = Date() + let ocr = ocrWindow(pid: app.processIdentifier) + if !ocr.isEmpty { lines = Array(NSOrderedSet(array: lines + ocr)) as? [String] ?? lines; via = "ocr" } + } + let ms = Int(Date().timeIntervalSince(started) * 1000) + let now = Date() + var fresh: [String] = [] + for line in lines { + let h = SHA256.hash(data: Data(line.utf8)).prefix(8).map { String(format: "%02x", $0) }.joined() + if let t = seen[h], now.timeIntervalSince(t) < 6 * 3600 { continue } + seen[h] = now + fresh.append(mask(line)) + } + if seen.count > 20000 { seen = seen.filter { now.timeIntervalSince($0.value) < 3600 } } + if !fresh.isEmpty || lines.isEmpty { note("\(bundle): \(lines.count) lines via \(via), \(fresh.count) new (\(ms)ms)") } + guard !fresh.isEmpty else { return } + append(["t": ISO8601DateFormatter().string(from: now), "app": app.localizedName ?? bundle, "bundle": bundle, + "window": title, "url": url ?? NSNull(), "via": via, "lines": fresh]) + if now.timeIntervalSince(lastPrune) > 24 * 3600 { prune() } + } + + /// Bounded BFS over the focused window collecting visible text. One batched AX round trip per node + /// (role, subrole, value, title, description, children), capped in nodes and characters, so a huge + /// tree (Electron, long web pages) costs a bounded few hundred ms, never seconds. Never reads secure fields. + private func readText(_ root: AXUIElement) -> [String] { + let attrs = [kAXRoleAttribute, kAXSubroleAttribute, kAXValueAttribute, kAXTitleAttribute, + kAXDescriptionAttribute, kAXChildrenAttribute] as CFArray + let textRoles: Set = ["AXStaticText", "AXTextArea", "AXTextField", "AXHeading", "AXLink", "AXCell"] + var queue: [AXUIElement] = [root], out: [String] = [], seenLocal = Set() + var visited = 0, chars = 0 + while !queue.isEmpty && visited < 1500 && chars < 20000 { + let el = queue.removeFirst(); visited += 1 + var raw: CFArray? + guard AXUIElementCopyMultipleAttributeValues(el, attrs, AXCopyMultipleAttributeOptions(rawValue: 0), &raw) == .success, + let vals = raw as? [Any], vals.count == 6 else { continue } + let role = vals[0] as? String ?? "", sub = vals[1] as? String ?? "" + if role == "AXSecureTextField" || sub == "AXSecureTextField" { continue } // never read passwords + if textRoles.contains(role) { + for v in vals[2...4] { + guard let str = v as? String, !str.isEmpty else { continue } + for piece in str.split(whereSeparator: \.isNewline) { + let line = piece.trimmingCharacters(in: .whitespaces) + guard line.count >= 3, line.count <= 2000, seenLocal.insert(line).inserted else { continue } + out.append(line); chars += line.count + } + } + } + if let kids = vals[5] as? [AXUIElement] { queue.append(contentsOf: kids) } + } + return out + } + + /// Capture ONLY the frontmost window of `pid` with the system screencapture tool (same path the app's + /// OCR already uses), recognise text on-device, delete the image immediately. Needs Screen Recording + /// permission; macOS shows its own prompt once, the user decides there. + private func ocrWindow(pid: pid_t) -> [String] { + guard CGPreflightScreenCaptureAccess() else { + if !askedScreenPermission { askedScreenPermission = true; note("OCR needs Screen Recording permission") + DispatchQueue.main.async { _ = CGRequestScreenCaptureAccess() } } + return [] + } + guard let infos = CGWindowListCopyWindowInfo([.optionOnScreenOnly, .excludeDesktopElements], kCGNullWindowID) as? [[String: Any]], + let info = infos.first(where: { ($0[kCGWindowOwnerPID as String] as? pid_t) == pid && ($0[kCGWindowLayer as String] as? Int) == 0 }), + let wid = info[kCGWindowNumber as String] as? Int else { return [] } + let tmpDir = (relay as NSString).appendingPathComponent("tmp") + try? FileManager.default.createDirectory(atPath: tmpDir, withIntermediateDirectories: true, attributes: [.posixPermissions: 0o700]) + let path = (tmpDir as NSString).appendingPathComponent("journal-ocr-\(UUID().uuidString).png") + defer { try? FileManager.default.removeItem(atPath: path) } + let cap = Process() + cap.executableURL = URL(fileURLWithPath: "/usr/sbin/screencapture") + cap.arguments = ["-x", "-o", "-l", String(wid), "-t", "png", path] + do { try cap.run(); cap.waitUntilExit() } catch { return [] } + guard var img = NSImage(contentsOfFile: path)?.cgImage(forProposedRect: nil, context: nil, hints: nil) else { return [] } + // Retina windows are ~3k px wide; text stays legible at 1600, and Vision's cost scales with pixels. + if img.width > 1600 { + let w = 1600, h = img.height * 1600 / img.width + if let ctx = CGContext(data: nil, width: w, height: h, bitsPerComponent: 8, bytesPerRow: 0, + space: CGColorSpaceCreateDeviceRGB(), bitmapInfo: CGImageAlphaInfo.noneSkipLast.rawValue) { + ctx.interpolationQuality = .medium + ctx.draw(img, in: CGRect(x: 0, y: 0, width: w, height: h)) + if let small = ctx.makeImage() { img = small } + } + } + var out: [String] = [] + let req = VNRecognizeTextRequest { r, _ in + for o in (r.results as? [VNRecognizedTextObservation]) ?? [] { + if let t = o.topCandidates(1).first?.string.trimmingCharacters(in: .whitespaces), t.count >= 3 { out.append(t) } + } + } + req.recognitionLevel = .accurate + req.usesLanguageCorrection = true + req.minimumTextHeight = 0.008 // skip hairline text (status bars, footers): fewer regions, faster + try? VNImageRequestHandler(cgImage: img, options: [:]).perform([req]) + return out + } + + private var lastNote = "" + private func note(_ s: String) { if s != lastNote { lastNote = s; godLog("journal: " + s) } } + + // MARK: storage + + private func append(_ entry: [String: Any]) { + let fm = FileManager.default + try? fm.createDirectory(atPath: dir, withIntermediateDirectories: true, attributes: [.posixPermissions: 0o700]) + let day = String(ISO8601DateFormatter().string(from: Date()).prefix(10)) + let path = (dir as NSString).appendingPathComponent("\(day).jsonl") + guard var data = try? JSONSerialization.data(withJSONObject: entry) else { return } + data.append(0x0a) + if !fm.fileExists(atPath: path) { fm.createFile(atPath: path, contents: nil, attributes: [.posixPermissions: 0o600]) } + if let h = FileHandle(forWritingAtPath: path) { h.seekToEndOfFile(); h.write(data); try? h.close() } + } + + private func prune() { + lastPrune = Date() + let keep = config().retentionDays + let cutoff = Date().addingTimeInterval(-Double(keep) * 86400) + let fmt = DateFormatter(); fmt.dateFormat = "yyyy-MM-dd"; fmt.timeZone = TimeZone(identifier: "UTC") + for f in (try? FileManager.default.contentsOfDirectory(atPath: dir)) ?? [] where f.hasSuffix(".jsonl") { + if let d = fmt.date(from: String(f.prefix(10))), d < cutoff { + try? FileManager.default.removeItem(atPath: (dir as NSString).appendingPathComponent(f)) + } + } + } + + private func config() -> (block: Set, blockWords: [String], retentionDays: Int) { + var block = Self.defaultBlock, words = Self.defaultBlockWords, days = 30 + let path = (relay as NSString).appendingPathComponent("journal.json") + if let d = FileManager.default.contents(atPath: path), + let j = try? JSONSerialization.jsonObject(with: d) as? [String: Any] { + (j["block"] as? [String])?.forEach { block.insert($0.lowercased()) } + (j["blockWords"] as? [String])?.forEach { words.append($0.lowercased()) } + if let r = j["retentionDays"] as? Int, r > 0 { days = r } + } + return (block, words, days) + } + + /// Mask secret-looking tokens before they ever touch disk (API keys, long hex/base64 blobs, card numbers). + private func mask(_ s: String) -> String { + var out = s + let patterns = [#"\b(sk|pk|rk|ghp|gho|xox[abp]|AKIA|AIza)[-_A-Za-z0-9]{2,}[•*…]*"#, + #"\b[A-Fa-f0-9]{32,}\b"#, #"\b[A-Za-z0-9+/]{40,}={0,2}"#, #"\b(?:\d[ -]?){13,19}\b"#] + for p in patterns { + out = out.replacingOccurrences(of: p, with: "[masked]", options: .regularExpression) + } + return out + } + + // MARK: AX helpers (own copies — AmbientSensor's are file-private and its contract stays separate) + + private func jString(_ el: AXUIElement, _ attr: String) -> String? { + var v: CFTypeRef? + guard AXUIElementCopyAttributeValue(el, attr as CFString, &v) == .success, let s = v as? String, + !s.isEmpty else { return nil } + return s + } + private func jElement(_ el: AXUIElement, _ attr: String) -> AXUIElement? { + var v: CFTypeRef? + guard AXUIElementCopyAttributeValue(el, attr as CFString, &v) == .success, let r = v, + CFGetTypeID(r) == AXUIElementGetTypeID() else { return nil } + return (r as! AXUIElement) + } + private func jChildren(_ el: AXUIElement) -> [AXUIElement] { + var v: CFTypeRef? + guard AXUIElementCopyAttributeValue(el, kAXChildrenAttribute as CFString, &v) == .success, + let arr = v as? [AXUIElement] else { return [] } + return arr + } + private func jURL(_ win: AXUIElement) -> String? { + var q: [AXUIElement] = [win], n = 0 + while !q.isEmpty && n < 300 { + let el = q.removeFirst(); n += 1 + if jString(el, kAXRoleAttribute as String) == "AXWebArea" { + var v: CFTypeRef? + if AXUIElementCopyAttributeValue(el, "AXURL" as CFString, &v) == .success, let u = v as? URL { return u.absoluteString } + } + q.append(contentsOf: jChildren(el)) + } + return nil + } +} diff --git a/packages/menubar/build.sh b/packages/menubar/build.sh index 9a65e90..3bf25f7 100755 --- a/packages/menubar/build.sh +++ b/packages/menubar/build.sh @@ -5,7 +5,7 @@ cd "$(dirname "$0")" echo "[menubar] compiling…" mkdir -p build -swiftc -O -o build/Relay main.swift RelayMenuBar.swift CursorGuide.swift TeamSection.swift TeamCursorsOverlay.swift IgnitionOverlay.swift DictationScratch.swift WhiteboardPanel.swift NotchTray.swift NotchLauncherView.swift LauncherRouting.swift ConsentRouting.swift GodWidgetKit.swift GodWebWindow.swift StoreFrontView.swift HtmlCapability.swift SkillRunner.swift AmbientSensor.swift AmbientCanvas.swift OSShellView.swift OSSurfaceWorkspace.swift OSSurfaceAutomate.swift OSSurfaceKnowledge.swift OSSurfaceDo.swift -framework AppKit -framework SwiftUI -framework WebKit -framework ApplicationServices -framework CoreServices +swiftc -O -o build/Relay main.swift RelayMenuBar.swift CursorGuide.swift TeamSection.swift TeamCursorsOverlay.swift IgnitionOverlay.swift DictationScratch.swift WhiteboardPanel.swift NotchTray.swift NotchLauncherView.swift LauncherRouting.swift ConsentRouting.swift GodWidgetKit.swift GodWebWindow.swift StoreFrontView.swift HtmlCapability.swift SkillRunner.swift AmbientSensor.swift AmbientCanvas.swift ScreenJournal.swift LooseNotch.swift OSShellView.swift OSSurfaceWorkspace.swift OSSurfaceAutomate.swift OSSurfaceKnowledge.swift OSSurfaceDo.swift -framework AppKit -framework SwiftUI -framework WebKit -framework ApplicationServices -framework CoreServices APP="Switchboard.app" rm -rf "$APP" diff --git a/packages/switchboard-mcp/install-watch.mjs b/packages/switchboard-mcp/install-watch.mjs new file mode 100644 index 0000000..3e39a27 --- /dev/null +++ b/packages/switchboard-mcp/install-watch.mjs @@ -0,0 +1,41 @@ +#!/usr/bin/env node +// Install / remove the screen-watch LaunchAgent (com.relay.screenwatch). Opt-in; the watcher itself only +// acts while ~/.relay/journal-on exists and a System 1 engine is available. +// node packages/switchboard-mcp/install-watch.mjs # install + start +// node packages/switchboard-mcp/install-watch.mjs --remove # stop + uninstall +import { writeFileSync, rmSync, existsSync } from 'node:fs'; +import { join, dirname } from 'node:path'; +import { homedir } from 'node:os'; +import { execFileSync } from 'node:child_process'; +import { fileURLToPath } from 'node:url'; + +const LABEL = 'com.relay.screenwatch'; +const plist = join(homedir(), 'Library/LaunchAgents', `${LABEL}.plist`); +const domain = `gui/${process.getuid()}`; +const quiet = args => { try { execFileSync('launchctl', args, { stdio: 'ignore' }); } catch {} }; + +quiet(['bootout', `${domain}/${LABEL}`]); +// bootout is asynchronous; bootstrapping before the old job is gone fails with "5: Input/output error". +for (let i = 0; i < 50; i++) { + try { execFileSync('launchctl', ['print', `${domain}/${LABEL}`], { stdio: 'ignore' }); } catch { break; } + execFileSync('sleep', ['0.2']); +} +if (process.argv.includes('--remove')) { rmSync(plist, { force: true }); console.log('screen watch removed'); process.exit(0); } + +const script = join(dirname(fileURLToPath(import.meta.url)), 'watch.mjs'); +const log = join(homedir(), '.relay/screen-watch.log'); +writeFileSync(plist, ` + + + Label${LABEL} + ProgramArguments${process.execPath}${script} + RunAtLoad + KeepAlive + ProcessTypeBackground + Nice10 + StandardOutPath${log} + StandardErrorPath${log} + +`); +execFileSync('launchctl', ['bootstrap', domain, plist]); +console.log(`screen watch installed → ${plist}\nlog: ${log}${existsSync(join(homedir(), '.relay/journal-on')) ? '' : '\n(journal is off: touch ~/.relay/journal-on to start recording)'}`); diff --git a/packages/switchboard-mcp/journal.mjs b/packages/switchboard-mcp/journal.mjs new file mode 100644 index 0000000..59ff451 --- /dev/null +++ b/packages/switchboard-mcp/journal.mjs @@ -0,0 +1,114 @@ +// Screen journal: read side + controls for ~/.relay/journal (written by packages/menubar/ScreenJournal.swift). +// Recall = jevgrep over the journal: a LOCAL keyword prefilter narrows days of screen text to a few hundred +// candidate lines, then System 1 (switchboard_decide's engine) asks "does this line help answer the query?" +// per line. Only those candidates leave the Mac, redacted (emails, phone numbers) first. With no System 1 +// engine, recall returns the local prefilter hits for the agent to read itself. +import { existsSync, readdirSync, readFileSync, writeFileSync, rmSync, statSync } from 'node:fs'; +import { join } from 'node:path'; +import { relayDir } from './presence.mjs'; +import { createSystem1 } from './system1.mjs'; + +const STOP = new Set('a an the and or but of to in on at for with from by about what which who whom when where why how did do does was were is are be been it this that these those i me my we our you your he she they them his her their any some there here said say says tell told'.split(' ')); +const tokens = s => s.toLowerCase().split(/[^a-z0-9]+/).filter(t => t.length > 2 && !STOP.has(t)); + +export function redact(s) { + return s.replace(/\b(sk|pk|rk|ghp|gho|xox[abp]|AKIA|AIza)[-_A-Za-z0-9]{2,}[•*…]*/g, '[key]') + .replace(/[\w.+-]+@[\w-]+\.[\w.-]+/g, '[email]') + .replace(/\+?\d[\d ()-]{8,}\d/g, '[number]'); +} + +export function createJournal({ dir = relayDir(), system1 = createSystem1() } = {}) { + const jdir = join(dir, 'journal'); + const flag = join(dir, 'journal-on'); + const days = () => (existsSync(jdir) ? readdirSync(jdir).filter(f => /^\d{4}-\d{2}-\d{2}\.jsonl$/.test(f)).sort() : []); + + function* entries({ sinceDays = 7 } = {}) { + const cutoff = new Date(Date.now() - sinceDays * 86400e3).toISOString().slice(0, 10); + for (const f of days().filter(f => f.slice(0, 10) >= cutoff).reverse()) { + const rows = readFileSync(join(jdir, f), 'utf8').split('\n').filter(Boolean); + for (let i = rows.length - 1; i >= 0; i--) { try { yield JSON.parse(rows[i]); } catch {} } + } + } + + function status() { + const files = days(); + const bytes = files.reduce((n, f) => n + statSync(join(jdir, f)).size, 0); + let last = null; for (const e of entries({ sinceDays: 2 })) { last = { t: e.t, app: e.app }; break; } + return { recording: existsSync(flag), days: files.length, oldest: files[0]?.slice(0, 10) ?? null, + megabytes: +(bytes / 1e6).toFixed(2), lastEntry: last, dir: jdir }; + } + + function setRecording(on) { + if (on) writeFileSync(flag, ''); else rmSync(flag, { force: true }); + return status(); + } + + /** Drop every entry newer than `minutes` ago (the "forget the last hour" button). */ + function forget(minutes) { + const cutoff = new Date(Date.now() - minutes * 60e3).toISOString(); + let removed = 0; + for (const f of days()) { + const p = join(jdir, f); + const rows = readFileSync(p, 'utf8').split('\n').filter(Boolean); + const keep = rows.filter(r => { try { return JSON.parse(r).t < cutoff; } catch { return true; } }); + removed += rows.length - keep.length; + if (keep.length !== rows.length) { + if (keep.length) writeFileSync(p, keep.join('\n') + '\n', { mode: 0o600 }); else rmSync(p); + } + } + return { removed, ...status() }; + } + + /** Local keyword prefilter → candidate lines with their context, best first. */ + function prefilter(query, { sinceDays = 7, app, max = 200 } = {}) { + const q = tokens(query); + const hits = []; + for (const e of entries({ sinceDays })) { + if (app && !`${e.app} ${e.bundle}`.toLowerCase().includes(app.toLowerCase())) continue; + const ctx = `${e.app} ${e.window ?? ''}`.toLowerCase(); + for (const line of e.lines ?? []) { + const lt = line.toLowerCase(); + let score = 0; + for (const t of q) if (lt.includes(t)) score += 2; else if (ctx.includes(t)) score += 1; + if (score > 0) hits.push({ score, line, t: e.t, app: e.app, window: e.window ?? '' }); + } + } + hits.sort((a, b) => b.score - a.score || (a.t < b.t ? 1 : -1)); + const seen = new Set(); + return hits.filter(h => !seen.has(h.line) && seen.add(h.line)).slice(0, max); + } + + async function recall(query, { sinceDays = 7, app, limit = 20, batch = 40 } = {}) { + const candidates = prefilter(query, { sinceDays, app }); + if (!candidates.length) return { query, engine: null, matches: [], note: 'No journal lines share words with the query.' }; + // Recall never cold-starts a local engine (a slow one would stall the answer and load the Mac): + // hosted when keyed, a local/custom server only if it is already up, else the local-hits fallback. + let engine = system1.engine(); + if (engine !== 'hosted' && !(await system1.up())) engine = null; + if (engine === 'hosted' && system1.keyOk && !(await system1.keyOk())) engine = null; // bad key: send nothing + if (!engine) { + return { query, engine: null, note: 'No System 1 engine; these are local keyword matches. Read them and judge relevance yourself.', + matches: candidates.slice(0, limit) }; + } + const scored = []; + try { + for (let i = 0; i < candidates.length; i += batch) { + const chunk = candidates.slice(i, i + batch); + const state = chunk.map((c, j) => `[${j}] (${c.app} · ${c.window.slice(0, 60)}) ${redact(c.line)}`).join('\n'); + const questions = Object.fromEntries(chunk.map((_, j) => [`l${j}`, { + type: 'noul', instructions: `Line [${j}] contains information that helps answer: "${query}"` }])); + const r = await system1.decide({ state, questions }); + chunk.forEach((c, j) => scored.push({ ...c, p: r.answers?.[`l${j}`]?.noul ?? 0 })); + engine = r.engine; + } + } catch (e) { + return { query, engine: null, note: `System 1 failed (${e.message.slice(0, 160)}); these are local keyword matches. Read them and judge relevance yourself.`, + matches: candidates.slice(0, limit) }; + } + scored.sort((a, b) => b.p - a.p); + return { query, engine, matches: scored.filter(s => s.p >= 0.5).slice(0, limit), + considered: candidates.length }; + } + + return { status, setRecording, forget, prefilter, recall }; +} diff --git a/packages/switchboard-mcp/package.json b/packages/switchboard-mcp/package.json index 1ec4801..b46f197 100644 --- a/packages/switchboard-mcp/package.json +++ b/packages/switchboard-mcp/package.json @@ -9,7 +9,7 @@ }, "scripts": { "start": "node switchboard-mcp.mjs mcp", - "test": "node switchboard-mcp.test.mjs && node --test presence.test.mjs", + "test": "node switchboard-mcp.test.mjs && node --test presence.test.mjs watch.test.mjs", "test:plugin": "node --test plugin.test.mjs" }, "dependencies": { diff --git a/packages/switchboard-mcp/switchboard-mcp.mjs b/packages/switchboard-mcp/switchboard-mcp.mjs index ec25c82..82e755a 100644 --- a/packages/switchboard-mcp/switchboard-mcp.mjs +++ b/packages/switchboard-mcp/switchboard-mcp.mjs @@ -26,6 +26,7 @@ import { withDaemon, daemonSbForOrigin, daemonAvailable, readPairingToken, WS_UR import { mockSb } from "./mock-sb.mjs"; import { scaffoldWrapp } from "./scaffold.mjs"; import { registerPresenceTools, relayDir, atomicJSON, appRunning } from "./presence.mjs"; +import { registerSystem1Tools } from "./system1.mjs"; let clientSource = "Switchboard agent"; // The task board is the one thing the connector touches by FILE, not daemon: tasks live as plain // `tasks.md` lines in the project's vault (the same dialect the OS board + Obsidian read). The pure @@ -344,6 +345,7 @@ async function main() { clientSource = /codex/i.test(name) ? "Codex" : /claude/i.test(name) ? "Claude Code" : name; }; registerPresenceTools(server); + registerSystem1Tools(server); const table = actionTable(); for (const [name, entry] of table) { diff --git a/packages/switchboard-mcp/system1.mjs b/packages/switchboard-mcp/system1.mjs new file mode 100644 index 0000000..d32d105 --- /dev/null +++ b/packages/switchboard-mcp/system1.mjs @@ -0,0 +1,140 @@ +// System 1: fast typed decisions from a local System One model (OneJev via packages/system1/serve.py). +// State + typed questions in, calibrated probabilities out, no generated text. The engine speaks TypeSafe's +// /v1/systemone contract, so SYSTEM1_URL can point at any compatible server (Kev, Von, hosted Jev). +// Device-lightness: the engine is opt-in (packages/system1/install.sh), starts on first use, unloads when idle. +import { existsSync, readFileSync, openSync } from 'node:fs'; +import { join, extname } from 'node:path'; +import { spawn } from 'node:child_process'; +import { z } from 'zod'; +import { relayDir } from './presence.mjs'; +import { createJournal } from './journal.mjs'; + +const MIME = { '.png': 'image/png', '.jpg': 'image/jpeg', '.jpeg': 'image/jpeg', '.webp': 'image/webp', '.gif': 'image/gif' }; +const sleep = ms => new Promise(r => setTimeout(r, ms)); + +// Hosted lane: TypeSafe's Jev through OpenRouter's Decisions API (same {model, state, questions} → {answers} shape). +// BYO key, never stored by us: OPENROUTER_API_KEY in the env, or ~/.relay/tool-secrets.json {"system1": {"OPENROUTER_API_KEY": …}} (0600). +const HOSTED_URL = 'https://openrouter.ai/api/alpha/decisions'; +const HOSTED_MODEL = process.env.SYSTEM1_HOSTED_MODEL || 'typesafe/jev-1.13'; + +function hostedKey(dir) { + if (process.env.OPENROUTER_API_KEY) return process.env.OPENROUTER_API_KEY; + try { return JSON.parse(readFileSync(join(dir, 'tool-secrets.json'), 'utf8'))?.system1?.OPENROUTER_API_KEY || null; } + catch { return null; } +} + +export function createSystem1({ dir = relayDir(), url = process.env.SYSTEM1_URL } = {}) { + const base = () => url || `http://127.0.0.1:${process.env.SYSTEM1_PORT || 8017}`; + const python = join(dir, 'system1-venv/bin/python'); + const serve = join(dir, 'system1/serve.py'); + // Engine order: explicit SYSTEM1_URL → hosted Jev when a key is present → the opt-in local engine. + const engine = () => (url ? 'custom' : hostedKey(dir) ? 'hosted' : 'local'); + + async function up() { + try { return (await fetch(`${base()}/health`, { signal: AbortSignal.timeout(1000) })).ok; } + catch { return false; } + } + async function ensure(waitMs = 120_000) { + if (await up()) return; + if (url) throw new Error(`System 1 server at ${url} is not answering.`); + if (!existsSync(python) || !existsSync(serve)) { + throw new Error('System 1 has no engine. Hosted (recommended): the user adds an OpenRouter key as OPENROUTER_API_KEY or in ~/.relay/tool-secrets.json under system1. Local (opt-in, needs a fast GPU): bash packages/system1/install.sh in the relay repo.'); + } + const log = openSync(join(dir, 'system1.log'), 'a'); + spawn(python, [serve], { detached: true, stdio: ['ignore', log, log] }).unref(); + for (const end = Date.now() + waitMs; Date.now() < end; await sleep(1000)) if (await up()) return; + throw new Error(`System 1 did not come up within ${waitMs / 1000}s; see ${join(dir, 'system1.log')}.`); + } + // Local image paths are inlined as data URIs so any file the agent can read works, not just a media root. + function inlineMedia(media) { + return media?.map(m => { + if (!m.path) return m; + const mime = MIME[extname(m.path).toLowerCase()]; + if (!mime) throw new Error(`Unsupported image type: ${m.path}`); + const { path, ...rest } = m; + return { ...rest, data: `data:${mime};base64,${readFileSync(path).toString('base64')}` }; + }); + } + async function decide({ state, questions, media, timeoutSeconds = 60 }) { + const which = engine(); + let target, headers = { 'content-type': 'application/json' }, model = 'jev-latest'; + if (which === 'hosted') { + if (media?.length) throw new Error('Hosted Jev takes text and structured state only (no images yet). Describe the image in the state instead.'); + target = HOSTED_URL; model = HOSTED_MODEL; + headers.authorization = `Bearer ${hostedKey(dir)}`; + } else { + await ensure(); + target = `${base()}/v1/systemone`; + } + const res = await fetch(target, { + method: 'POST', headers, + body: JSON.stringify({ model, state, questions, media: which === 'hosted' ? undefined : inlineMedia(media) }), + signal: AbortSignal.timeout(timeoutSeconds * 1000), + }); + const text = await res.text(); + let body; try { body = JSON.parse(text); } catch { body = { detail: text.slice(0, 300) }; } + if (!res.ok) throw new Error(`System 1 (${which}) rejected the request (${res.status}): ${JSON.stringify(body.error ?? body.detail ?? body)}`); + const { qev, ...answer } = body; + return { engine: which, ...answer }; + } + // Hosted: confirm the key with OpenRouter (no user data sent) before anything else goes out. Cached 1h. + let verified = { at: 0, ok: false }; + async function keyOk() { + if (engine() !== 'hosted') return true; + if (Date.now() - verified.at < 3600e3) return verified.ok; + let ok = false; + try { ok = (await fetch('https://openrouter.ai/api/v1/key', { headers: { authorization: `Bearer ${hostedKey(dir)}` }, + signal: AbortSignal.timeout(10000) })).ok; } catch {} + verified = { at: Date.now(), ok }; + return ok; + } + return { up, ensure, decide, engine, keyOk }; +} + +const question = z.object({ + type: z.enum(['noul', 'choice', 'score']), + instructions: z.string().min(1), + criteria: z.union([z.record(z.string().nullable()), z.array(z.string())]).optional(), +}).passthrough(); + +export function registerSystem1Tools(server) { + const s1 = createSystem1(); + server.registerTool('switchboard_decide', { + description: 'System 1: fast, local, typed judgment with calibrated probabilities and no generated text. Give a state (text or object; images via media + placeholders) and named questions: noul (yes/no → probability), choice (criteria map → one option), score (ordered criteria list → level). Use it for VOLUME (triage hundreds of grep hits, files, log lines, screenshots before reading them) or cheap gating; for a handful of items just read them yourself. Treat confidence < 0.7 as "unsure" and check with your own reasoning. The result names its engine: "hosted" = TypeSafe Jev via OpenRouter (the state LEAVES this Mac, so never send secrets, keys or private user data), "local"/"custom" = on-device.', + inputSchema: { + state: z.union([z.string(), z.record(z.any())]).describe('What to judge. Reference media as , … in order.'), + questions: z.record(question).describe('Name → question. Questions share the state but cannot see each other.'), + media: z.array(z.object({ type: z.enum(['image', 'video']) }).passthrough()).optional() + .describe('[{type:"image", path|url|data}] — local paths are inlined automatically.'), + timeoutSeconds: z.number().int().min(1).max(600).optional(), + }, + annotations: { readOnlyHint: true }, + }, async args => { + try { return { content: [{ type: 'text', text: JSON.stringify(await s1.decide(args)) }] }; } + catch (e) { return { isError: true, content: [{ type: 'text', text: JSON.stringify({ ok: false, error: e.message }) }] }; } + }); + + const journal = createJournal({ system1: s1 }); + server.registerTool('switchboard_recall', { + description: 'Recall what the user saw on screen (the local screen journal, ~/.relay/journal) by meaning — jevgrep over their own history. A local keyword prefilter picks candidate lines, then System 1 scores each for relevance; returns matching lines with time, app and window so you can answer and cite. Phrase the query with the words likely to appear on screen (names, project, topic). Candidates (redacted) leave the Mac only when the engine is hosted. With no engine it returns local keyword hits for you to judge.', + inputSchema: { + query: z.string().min(2), sinceDays: z.number().int().min(1).max(90).optional(), + app: z.string().optional().describe('Only lines from this app (name or bundle id substring).'), + limit: z.number().int().min(1).max(100).optional(), + }, + annotations: { readOnlyHint: true }, + }, async ({ query, ...opts }) => { + try { return { content: [{ type: 'text', text: JSON.stringify(await journal.recall(query, opts)) }] }; } + catch (e) { return { isError: true, content: [{ type: 'text', text: JSON.stringify({ ok: false, error: e.message }) }] }; } + }); + server.registerTool('switchboard_journal', { + description: 'Control the local screen journal: status (recording? size, last entry), on, off, or forget (delete everything from the last N minutes). Recording is opt-in; only turn it on or off when the user asks.', + inputSchema: { action: z.enum(['status', 'on', 'off', 'forget']), minutes: z.number().int().min(1).optional() }, + }, async ({ action, minutes }) => { + try { + const out = action === 'status' ? journal.status() : action === 'on' ? journal.setRecording(true) + : action === 'off' ? journal.setRecording(false) : journal.forget(minutes ?? 60); + return { content: [{ type: 'text', text: JSON.stringify(out) }] }; + } catch (e) { return { isError: true, content: [{ type: 'text', text: JSON.stringify({ ok: false, error: e.message }) }] }; } + }); +} diff --git a/packages/switchboard-mcp/watch-replay.mjs b/packages/switchboard-mcp/watch-replay.mjs new file mode 100644 index 0000000..ece006c --- /dev/null +++ b/packages/switchboard-mcp/watch-replay.mjs @@ -0,0 +1,64 @@ +#!/usr/bin/env node +// Rapid trigger testing for screen watch: replay journal windows (+ labelled probe cases) through System 1 +// with both question sets and gates, and print what WOULD have fired. Rate caps/cooldowns are ignored so +// every trigger shows. Costs ~$0.00002 per window on hosted Jev; state is redacted like the live watcher. +// node packages/switchboard-mcp/watch-replay.mjs [--last 40] [--probes-only] +import { readFileSync, readdirSync, existsSync } from 'node:fs'; +import { join } from 'node:path'; +import { relayDir } from './presence.mjs'; +import { createSystem1 } from './system1.mjs'; +import { redact } from './journal.mjs'; +import { questionsFor, questionsV2, evaluate } from './watch.mjs'; + +// Probes with the verdict we WANT (card / memory / quiet), to catch a gate that goes deaf. +const PROBES = [ + { want: 'card', app: 'Slack', window: '#launch', lines: ['Mike Chen 10:42 AM', 'hey, can you send me the pitch deck before Friday? investors asked', 'Reply in thread'] }, + { want: 'card', app: 'Mail', window: 'Inbox', lines: ['From: Priya Rao', 'Subject: Intro to Sequoia scout', 'Happy to intro you to Arjun at Sequoia, want me to set it up this week?'] }, + { want: 'card', app: 'Google Chrome', window: 'Calendar', lines: ['Invitation: Design review with Asha', 'Thu 3pm to 4pm', 'Going? Yes No Maybe'] }, + { want: 'memory', app: 'Slack', window: 'DM Rahul', lines: ['Rahul: fyi I only take calls after 4pm IST, mornings are school runs'] }, + { want: 'quiet', app: 'UserNotificationCenter', window: '', lines: ['Allow "Google Chrome" to find devices on local networks?', 'This will allow you to select from available devices'] }, + { want: 'quiet', app: 'Claude', window: 'Claude', lines: ['in jev running rn? I can see it watching. but havent seen any card'] }, + { want: 'quiet', app: 'Google Chrome', window: 'Settings', lines: ['API Keys', 'sk-or-v1-29b•', 'Create key', 'Usage this month $0.02'] }, + { want: 'quiet', app: 'Google Chrome', window: 'YouTube', lines: ['How to cook biryani in 10 minutes', '1.2M views', 'Subscribe'] }, +]; + +function windows(dir, last) { + const jdir = join(dir, 'journal'); + if (!existsSync(jdir)) return []; + const rows = readdirSync(jdir).filter(f => f.endsWith('.jsonl')).sort().flatMap(f => + readFileSync(join(jdir, f), 'utf8').split('\n').filter(Boolean).map(r => { try { return JSON.parse(r); } catch { return null; } }).filter(Boolean)); + const groups = []; + for (const e of rows) { // merge like the watcher's settle window + const g = groups.at(-1); + if (g && g.bundle === e.bundle && g.window === e.window && Date.parse(e.t) - Date.parse(g.t) < 15e3) { g.lines.push(...e.lines); g.t = e.t; } + else groups.push({ ...e, lines: [...e.lines] }); + } + return groups.filter(g => g.lines.join(' ').length >= 20).slice(-last); +} + +const verdict = out => out.kind === 'card' ? `CARD ${out.sub}` : out.kind === 'memory' ? 'MEMORY' : out.kind === 'noticed' ? `noticed ${out.sub}` : '·'; + +async function main() { + const args = process.argv.slice(2); + const last = Number(args[args.indexOf('--last') + 1]) || 40; + const s1 = createSystem1(); + if (!(await s1.keyOk?.())) { console.error('No valid System 1 engine (hosted key).'); process.exit(1); } + const cases = [...PROBES.map(p => ({ ...p, probe: true })), ...(args.includes('--probes-only') ? [] : windows(relayDir(), last))]; + const tally = { v1: 0, v2: 0 }, misses = []; + const ctx = { now: Date.now(), recentCards: [], seenKeys: new Set(), muted: new Set() }; + for (const c of cases) { + const lines = c.lines.slice(-60).map(redact); + const r = await s1.decide({ state: { app: c.app, window: c.window ?? '', text: lines.join('\n') }, questions: questionsV2(lines) }); + const a = r.answers; + const v1 = evaluate(a, { ...ctx, lines, app: c.app }), v2 = evaluate(a, { ...ctx, lines, app: c.app, strict: true }); + if (v1.kind === 'card') tally.v1++; if (v2.kind === 'card') tally.v2++; + const got2 = v2.kind === 'card' ? 'card' : v2.kind === 'memory' ? 'memory' : 'quiet'; + if (c.probe && got2 !== c.want) misses.push(`${c.app}: wanted ${c.want}, v2 gave ${got2}`); + const n = q => (a?.[q]?.noul ?? 0).toFixed(2); + console.log(`${c.probe ? `probe(${c.want})`.padEnd(14) : (c.t ?? '').slice(11, 16).padEnd(14)} ${String(c.app).slice(0, 14).padEnd(14)} ` + + `${a.kind.choice.padEnd(11)} ${(a.kind.confidence ?? 0).toFixed(2)} person ${n('fromPerson')} ui ${n('systemUI')} own ${n('userOwn')} ` + + `v1 ${verdict(v1).padEnd(16)} v2 ${verdict(v2).padEnd(16)} | ${(v2.line ?? v1.line ?? lines[0] ?? '').slice(0, 50)}`); + } + console.log(`\n${cases.length} windows · cards v1 ${tally.v1} → v2 ${tally.v2}${misses.length ? `\nprobe misses (v2):\n ${misses.join('\n ')}` : '\nall probes as wanted under v2'}`); +} +main().catch(e => { console.error(e.message); process.exit(1); }); diff --git a/packages/switchboard-mcp/watch.mjs b/packages/switchboard-mcp/watch.mjs new file mode 100644 index 0000000..c328ea9 --- /dev/null +++ b/packages/switchboard-mcp/watch.mjs @@ -0,0 +1,202 @@ +#!/usr/bin/env node +// Screen watch — the LIVE layer over the screen journal (packages/menubar/ScreenJournal.swift → ~/.relay/journal). +// For each settled window change, one System 1 call classifies it: task / decision / action / opportunity / +// memory / nothing, how urgent, and which line is the point. That single call is both the live trigger +// (notch cards) and "jevtag" (memory.md in the vault). Everything else stays in the journal for recall. +// +// Guard rails: follows ~/.relay/journal-on (no journal → nothing to watch); never cold-starts a local engine; +// state is redacted before it leaves the Mac; at most MAX_PER_HOUR cards, one per app per APP_COOLDOWN, +// never the same line twice; muted apps skipped; engine errors back off. Every card outcome is logged as a +// label (screen-watch-labels.jsonl) so thresholds can be tuned on real answers. +// node packages/switchboard-mcp/watch.mjs # long-running (installed as a LaunchAgent) +import { existsSync, readFileSync, appendFileSync, writeFileSync, statSync, openSync, readSync, closeSync, watch, mkdirSync } from 'node:fs'; +import { join } from 'node:path'; +import { homedir } from 'node:os'; +import { createHash } from 'node:crypto'; +import { relayDir, createPresence } from './presence.mjs'; +import { createSystem1 } from './system1.mjs'; +import { redact } from './journal.mjs'; +import { addTask } from '../bank-mcp/tasks.mjs'; + +const MAX_PER_HOUR = 4; + +// Poll one card's result until answered or `waitSeconds` pass (own copy: keeps the watcher independent of +// presence.mjs's newer helpers). +async function waitForResult(presence, { surface, runId, waitSeconds, intervalMs = 1500 }) { + const deadline = Date.now() + waitSeconds * 1000; + for (;;) { + const r = presence.result({ surface, runId }); + if (r?.state !== 'pending' || Date.now() >= deadline) return r; + await new Promise(res => setTimeout(res, intervalMs)); + } +} +const APP_COOLDOWN_MS = 15 * 60e3; +const SETTLE_MS = 8e3; +const MIN_CONF = 0.6; + +export const KINDS = { + task: 'something the user needs to do later (a request, follow-up, deadline)', + decision: 'someone is waiting for the user to decide, approve or answer', + action: 'something an AI assistant could do for the user right now', + opportunity: 'a chance worth acting on: a lead, intro, offer, invite, opening', + memory: 'a fact worth remembering: a commitment, preference, person detail or project decision', + nothing: 'routine content: nothing to act on or remember', +}; + +export function questionsFor(lines) { + const numbered = Object.fromEntries(lines.map((l, i) => [`L${i}`, l.slice(0, 200)])); + return { + kind: { type: 'choice', instructions: 'What, if anything, on this screen matters to the user?', criteria: KINDS }, + urgency: { type: 'score', instructions: 'How soon does the user need to know?', criteria: ['can wait', 'today', 'right now'] }, + line: { type: 'choice', instructions: 'Which line is the key item?', criteria: numbered }, + }; +} + +// v2 (2026-09-29, after the first real cards were all noise: a macOS permission dialog, a masked key on a +// settings page, and the user's own messages to Claude): three extra yes/no reads decide WHO the text is +// from. A card needs a real person asking something of the user, not UI and not the user's own words. +export function questionsV2(lines) { + return { + ...questionsFor(lines), + fromPerson: { type: 'noul', instructions: 'A real person other than the user wrote this (a chat message, email, comment or invite) and it is addressed to the user or needs something from them.' }, + systemUI: { type: 'noul', instructions: 'The important text is software interface: a permission prompt, dialog, settings or account page, menu, or app navigation.' }, + userOwn: { type: 'noul', instructions: 'The important text was written by the user themselves: their own message, note, or instruction to an AI assistant.' }, + }; +} +const V2_CONF = 0.75; + +/** Pure gate: System 1 answers + context → what to do. Exported for tests. `strict` = the v2 who-wrote-it gate. */ +export function evaluate(answers, { lines, app, now, recentCards, seenKeys, muted, strict = false }) { + const kind = answers?.kind?.choice, conf = answers?.kind?.confidence ?? 0; + if (!kind || kind === 'nothing' || conf < MIN_CONF) return { kind: 'nothing' }; + if (strict) { + const p = q => answers?.[q]?.noul ?? 0; + const person = p('fromPerson') >= 0.85; // clearly a real person asking → easier bar, no urgency gate + if (conf < (person ? 0.65 : V2_CONF) || p('systemUI') >= 0.5 || p('userOwn') >= 0.5) return { kind: 'nothing', why: 'ui/own/low' }; + if (kind !== 'memory' && p('fromPerson') < 0.7) return { kind: 'nothing', why: 'not from a person' }; + } + const idx = Number(String(answers?.line?.choice ?? 'L0').slice(1)); + const line = lines[idx] ?? lines[0]; + const key = createHash('sha256').update(line).digest('hex').slice(0, 16); + if (seenKeys.has(key)) return { kind: 'seen' }; + if (kind === 'memory') return { kind, line, key, conf }; + if (strict && kind === 'action') return { kind: 'noticed', sub: kind, line, key, conf }; // actions: quiet until proven + const urgency = answers?.urgency?.probabilities ? Object.entries(answers.urgency.probabilities).sort((a, b) => b[1] - a[1])[0][0] : '0'; + const personAsk = strict && (answers?.fromPerson?.noul ?? 0) >= 0.85; + if (kind !== 'decision' && !personAsk && Number(urgency) < 1) return { kind: 'noticed', sub: kind, line, key, conf }; + if (muted.has(app)) return { kind: 'muted', sub: kind, line, key }; + const hourAgo = now - 3600e3; + if (recentCards.filter(c => c.t > hourAgo).length >= MAX_PER_HOUR) return { kind: 'capped', sub: kind, line, key, conf }; + if (recentCards.some(c => c.app === app && c.t > now - APP_COOLDOWN_MS)) return { kind: 'cooldown', sub: kind, line, key, conf }; + return { kind: 'card', sub: kind, line, key, conf }; +} + +const OPTIONS = { + task: [{ id: 'add', label: 'Add to board', recommended: true }, { id: 'dismiss', label: 'Not a task' }, { id: 'mute', label: 'Mute this app today' }], + opportunity: [{ id: 'add', label: 'Add to board', recommended: true }, { id: 'dismiss', label: 'Skip' }, { id: 'mute', label: 'Mute this app today' }], + decision: [{ id: 'add', label: 'Remind me (board)', recommended: true }, { id: 'dismiss', label: 'Handled' }, { id: 'mute', label: 'Mute this app today' }], + action: [{ id: 'claude', label: 'Hand to Claude', detail: 'Adds a card for Claude on the board', recommended: true }, { id: 'dismiss', label: 'No' }, { id: 'mute', label: 'Mute this app today' }], +}; +const TITLES = { task: 'Task spotted', opportunity: 'Opportunity', decision: 'Waiting on you', action: 'Claude could do this' }; + +export function createWatcher({ dir = relayDir(), vault = process.env.SWITCHBOARD_VAULT || join(homedir(), 'SwitchboardBrain'), + system1 = createSystem1(), presence = createPresence(), log = (...a) => console.log(new Date().toISOString(), ...a) } = {}) { + const jdir = join(dir, 'journal'), flag = join(dir, 'journal-on'); + const offsets = new Map(), pending = new Map(); + const recentCards = [], seenKeys = new Set(), muted = new Set(); + let backoffUntil = 0; + + const labels = rec => appendFileSync(join(dir, 'screen-watch-labels.jsonl'), JSON.stringify(rec) + '\n', { mode: 0o600 }); + const noticed = rec => appendFileSync(join(dir, 'screen-noticed.jsonl'), JSON.stringify(rec) + '\n', { mode: 0o600 }); + + function remember(entry, line, conf) { + if (!existsSync(vault)) mkdirSync(vault, { recursive: true }); + const p = join(vault, 'memory.md'); + if (!existsSync(p)) writeFileSync(p, '# Memory\n\nNoticed on screen by Switchboard (System 1). Edit or delete freely.\n\n'); + appendFileSync(p, `- ${entry.t.slice(0, 16).replace('T', ' ')} · ${entry.app}${entry.window ? ` · ${entry.window.slice(0, 50)}` : ''} — ${line} \n`); + } + + function fileTask(text, detail) { + const p = join(vault, 'tasks.md'); + const doc = existsSync(p) ? readFileSync(p, 'utf8') : ''; + const r = addTask(text, { detail }, doc); + if (r.added) writeFileSync(p, r.doc); + return r.added; + } + + async function handleCard(entry, out) { + const t = Date.now(); + recentCards.push({ t, app: entry.app }); + const submitted = presence.present({ + surface: 'guide', title: `${TITLES[out.sub]} · ${entry.app}`, source: 'Switchboard · screen watch', sourceId: 'screen-watch', + steps: [{ id: 'card', text: out.line.slice(0, 140), placement: 'notch', options: OPTIONS[out.sub], + ...(out.sub === 'decision' ? { say: `Someone's waiting on you in ${entry.app}.` } : {}) }], + }); + const r = await waitForResult(presence, { surface: 'guide', runId: submitted.runId, waitSeconds: 600 }); + const step = r.result?.results?.[0] ?? {}; + const choice = step.feedback?.note ? 'note' : step.chosenOption ?? 'none'; + const where = `${entry.app}${entry.window ? ' · ' + entry.window.slice(0, 50) : ''}`; + if (choice === 'add') fileTask(out.line.slice(0, 120), `From screen watch (${where}, ${entry.t.slice(0, 16)})`); + if (choice === 'claude') fileTask(`Claude: ${out.line.slice(0, 110)}`, `Spotted on screen (${where}). Hand to Claude.`); + if (choice === 'mute') muted.add(entry.app); + labels({ t: entry.t, app: entry.app, kind: out.sub, conf: out.conf, line: out.line, choice, note: step.feedback?.note ?? null }); + } + + async function evaluateEntry(entry) { + if (Date.now() < backoffUntil) return; + const engine = system1.engine(); + if (engine !== 'hosted' && !(await system1.up())) return; // never cold-start a local engine + if (system1.keyOk && !(await system1.keyOk())) { log('hosted key not valid; nothing sent'); backoffUntil = Date.now() + 3600e3; return; } + const lines = entry.lines.slice(-60); + if (lines.join(' ').length < 20) return; + const state = { app: entry.app, window: entry.window ?? '', text: lines.map(l => redact(l)).join('\n') }; + let r; + try { r = await system1.decide({ state, questions: questionsV2(lines.map(redact)) }); } + catch (e) { + const rejected = /\((401|403)\)/.test(e.message); // bad key: don't keep sending screen text to it + backoffUntil = Date.now() + (rejected ? 6 * 3600e3 : 10 * 60e3); + log(`System 1 ${rejected ? 'rejected the key' : 'error'}, backing off ${rejected ? '6h' : '10 min'}:`, e.message.slice(0, 160)); + return; + } + const out = evaluate(r.answers, { lines, app: entry.app, now: Date.now(), recentCards, seenKeys, muted, strict: true }); + if (out.key) seenKeys.add(out.key); + log(entry.app, '→', out.kind, out.sub ?? '', out.conf?.toFixed?.(2) ?? '', (out.line ?? '').slice(0, 60)); + if (out.kind === 'memory') remember(entry, out.line, out.conf); + else if (out.kind === 'card') handleCard(entry, out).catch(e => log('card error:', e.message)); + else if (['noticed', 'capped', 'cooldown', 'muted'].includes(out.kind)) noticed({ t: entry.t, app: entry.app, ...out }); + } + + // Settle: merge new lines per window for SETTLE_MS, then evaluate once. + function queue(entry) { + const k = `${entry.bundle}|${entry.window ?? ''}`; + const cur = pending.get(k); + if (cur) { clearTimeout(cur.timer); cur.entry.lines.push(...entry.lines); cur.entry.t = entry.t; } + const e = cur?.entry ?? { ...entry, lines: [...entry.lines] }; + pending.set(k, { entry: e, timer: setTimeout(() => { pending.delete(k); evaluateEntry(e).catch(err => log('eval error:', err.message)); }, SETTLE_MS) }); + } + + function readNew(file) { + const p = join(jdir, file); + let size; try { size = statSync(p).size; } catch { return; } + const from = offsets.get(file) ?? size; // start at the end: only react to what's new + if (size <= from) { offsets.set(file, size); return; } + const buf = Buffer.alloc(size - from), fd = openSync(p, 'r'); + readSync(fd, buf, 0, buf.length, from); closeSync(fd); + offsets.set(file, size); + for (const row of buf.toString('utf8').split('\n').filter(Boolean)) { + try { const e = JSON.parse(row); if (e.lines?.length) queue(e); } catch {} + } + } + + function start() { + if (!existsSync(jdir)) mkdirSync(jdir, { recursive: true, mode: 0o700 }); + const today = () => `${new Date().toISOString().slice(0, 10)}.jsonl`; + readNew(today()); + watch(jdir, (_, f) => { if (f?.endsWith('.jsonl') && existsSync(flag)) readNew(f); }); + log('screen watch started; engine:', system1.engine()); + } + + return { start, evaluateEntry, queue }; +} + +if (import.meta.url === `file://${process.argv[1]}`) createWatcher().start(); diff --git a/packages/switchboard-mcp/watch.test.mjs b/packages/switchboard-mcp/watch.test.mjs new file mode 100644 index 0000000..fa47ffc --- /dev/null +++ b/packages/switchboard-mcp/watch.test.mjs @@ -0,0 +1,81 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { mkdtempSync, readFileSync, existsSync } from 'node:fs'; +import { join } from 'node:path'; +import { tmpdir } from 'node:os'; +import { evaluate, createWatcher, questionsFor, questionsV2 } from './watch.mjs'; + +const ans = (kind, conf, line = 'L0', urgency = { 0: 0.1, 1: 0.2, 2: 0.7 }) => + ({ kind: { choice: kind, confidence: conf }, line: { choice: line }, urgency: { probabilities: urgency } }); +const ctx = (o = {}) => ({ lines: ['Mike: can you send the deck by Friday?', 'ok'], app: 'Slack', now: 1e12, + recentCards: [], seenKeys: new Set(), muted: new Set(), ...o }); + +test('nothing and low confidence never raise', () => { + assert.equal(evaluate(ans('nothing', 0.99), ctx()).kind, 'nothing'); + assert.equal(evaluate(ans('task', 0.4), ctx()).kind, 'nothing'); +}); + +test('urgent task raises a card on the chosen line', () => { + const out = evaluate(ans('task', 0.8), ctx()); + assert.equal(out.kind, 'card'); assert.equal(out.sub, 'task'); assert.match(out.line, /deck/); +}); + +test('non-urgent task is only noticed; decisions raise regardless of urgency', () => { + const calm = { 0: 0.9, 1: 0.05, 2: 0.05 }; + assert.equal(evaluate(ans('task', 0.8, 'L0', calm), ctx()).kind, 'noticed'); + assert.equal(evaluate(ans('decision', 0.8, 'L0', calm), ctx()).kind, 'card'); +}); + +test('memory goes to memory, not a card', () => assert.equal(evaluate(ans('memory', 0.9), ctx()).kind, 'memory')); + +test('rate cap, per-app cooldown, mute and dedupe', () => { + const now = 1e12; + const four = Array.from({ length: 4 }, (_, i) => ({ t: now - (i + 1) * 60e3, app: `A${i}` })); + assert.equal(evaluate(ans('task', 0.8), ctx({ recentCards: four })).kind, 'capped'); + assert.equal(evaluate(ans('task', 0.8), ctx({ recentCards: [{ t: now - 60e3, app: 'Slack' }] })).kind, 'cooldown'); + assert.equal(evaluate(ans('task', 0.8), ctx({ muted: new Set(['Slack']) })).kind, 'muted'); + const first = evaluate(ans('task', 0.8), ctx()); + assert.equal(evaluate(ans('task', 0.8), ctx({ seenKeys: new Set([first.key]) })).kind, 'seen'); +}); + +test('questions number the lines for the key-line choice', () => { + const q = questionsFor(['a line', 'b line']); + assert.deepEqual(Object.keys(q.line.criteria), ['L0', 'L1']); +}); + +test('end to end: memory lands in vault memory.md, card adds a board task, state is redacted', async () => { + const dir = mkdtempSync(join(tmpdir(), 'sw-')), vault = mkdtempSync(join(tmpdir(), 'vault-')); + const sent = []; + const system1 = { engine: () => 'hosted', up: async () => true, + decide: async ({ state }) => { sent.push(state.text); + const person = { fromPerson: { noul: 0.9 }, systemUI: { noul: 0.05 }, userOwn: { noul: 0.05 } }; + return { answers: { ...(state.text.includes('prefers') ? ans('memory', 0.9) : ans('task', 0.9)), ...person } }; } }; + let presented; + const presence = { present: p => { presented = p; return { runId: 'r1' }; }, + result: () => ({ ok: true, state: 'answered', result: { results: [{ chosenOption: 'add' }] } }) }; + const w = createWatcher({ dir, vault, system1, presence, log: () => {} }); + await w.evaluateEntry({ t: '2026-09-29T10:00:00Z', app: 'Mail', window: 'Inbox', lines: ['Priya prefers calls after 4pm, priya@x.com'] }); + assert.match(readFileSync(join(vault, 'memory.md'), 'utf8'), /Priya prefers calls/); + assert.ok(!sent[0].includes('priya@x.com'), 'email redacted before leaving the Mac'); + await w.evaluateEntry({ t: '2026-09-29T10:05:00Z', app: 'Slack', window: '#launch', lines: ['Mike: can you send the deck by Friday?'] }); + await new Promise(r => setTimeout(r, 50)); + assert.equal(presented.title, 'Task spotted · Slack'); + assert.ok(existsSync(join(vault, 'tasks.md'))); + assert.match(readFileSync(join(vault, 'tasks.md'), 'utf8'), /send the deck by Friday/); + assert.match(readFileSync(join(dir, 'screen-watch-labels.jsonl'), 'utf8'), /"choice":"add"/); +}); + +const who = (person, ui, own) => ({ fromPerson: { noul: person }, systemUI: { noul: ui }, userOwn: { noul: own } }); +test('strict: UI dialogs, the user\'s own words, and non-person text never raise', () => { + assert.equal(evaluate({ ...ans('decision', 0.99), ...who(0.1, 0.95, 0.04) }, ctx({ strict: true })).kind, 'nothing'); + assert.equal(evaluate({ ...ans('decision', 0.97), ...who(0.4, 0.6, 0.6) }, ctx({ strict: true })).kind, 'nothing'); + assert.equal(evaluate({ ...ans('task', 0.9), ...who(0.5, 0.1, 0.1) }, ctx({ strict: true })).kind, 'nothing'); +}); +test('strict: a clear person ask raises even when "can wait"; actions stay quiet', () => { + const calm = { 0: 0.9, 1: 0.05, 2: 0.05 }; + assert.equal(evaluate({ ...ans('task', 0.99, 'L0', calm), ...who(0.91, 0.08, 0.2) }, ctx({ strict: true })).kind, 'card'); + assert.equal(evaluate({ ...ans('opportunity', 0.69), ...who(0.94, 0.06, 0.08) }, ctx({ strict: true })).kind, 'card'); + assert.equal(evaluate({ ...ans('action', 0.9), ...who(0.9, 0.1, 0.1) }, ctx({ strict: true })).kind, 'noticed'); +}); +test('v2 adds the three who-wrote-it questions', () => assert.deepEqual( + Object.keys(questionsV2(['x'])).slice(-3), ['fromPerson', 'systemUI', 'userOwn'])); diff --git a/packages/system1/install.sh b/packages/system1/install.sh new file mode 100644 index 0000000..73df321 --- /dev/null +++ b/packages/system1/install.sh @@ -0,0 +1,16 @@ +#!/usr/bin/env bash +# Opt-in install of the System 1 engine (OneJev via qev) into ~/.relay. Nothing runs until asked: +# serve.py starts on demand and unloads after SYSTEM1_IDLE_S idle seconds. +# bash packages/system1/install.sh # then the first request downloads the weights (~1.7GB for 0.8B) +set -euo pipefail +HERE="$(cd "$(dirname "$0")" && pwd)" +DIR="${RELAY_DIR:-$HOME/.relay}" +QEV_REF="90a4584f17b42b8802da24324eb5318a45eb6e6b" # OmniJev/OneJev, pinned + +command -v uv >/dev/null || { echo "needs uv: https://docs.astral.sh/uv/" >&2; exit 1; } +[ -x "$DIR/system1-venv/bin/python" ] || uv venv --python 3.13 "$DIR/system1-venv" +VIRTUAL_ENV="$DIR/system1-venv" UV_HTTP_TIMEOUT=300 \ + uv pip install "qev @ git+https://github.com/OmniJev/OneJev@$QEV_REF" pillow torchvision +mkdir -p "$DIR/system1" +cp "$HERE/serve.py" "$DIR/system1/serve.py" +echo "System 1 installed. Start on demand: $DIR/system1-venv/bin/python $DIR/system1/serve.py" diff --git a/packages/system1/serve.py b/packages/system1/serve.py new file mode 100644 index 0000000..d0b29f9 --- /dev/null +++ b/packages/system1/serve.py @@ -0,0 +1,84 @@ +#!/usr/bin/env python3 +"""Switchboard System 1 engine: a local TypeSafe-compatible /v1/systemone server that unloads when idle. + +Wraps `qev serve` (OneJev, https://github.com/OmniJev/OneJev) so the model only occupies RAM while something is +asking: it starts on demand (switchboard_decide, the notch stop hook) and exits after SYSTEM1_IDLE_S seconds with +no request. Device-lightness rule: a store capability must never slow the machine while unused. + +Env: SYSTEM1_MODEL (default onejev-0.8b), SYSTEM1_PORT (8017), SYSTEM1_IDLE_S (600), SYSTEM1_DEVICE (mps|cpu). +Writes ~/.relay/system1.json {url, model, pid, startedAt} while alive so clients can find it. +""" +import json +import os +import sys +import threading +import time + +os.environ.setdefault("PYTORCH_ENABLE_MPS_FALLBACK", "1") + +import qev.server as server # noqa: E402 + +RELAY = os.environ.get("RELAY_DIR", os.path.expanduser("~/.relay")) +STATE = os.path.join(RELAY, "system1.json") +MODEL = os.environ.get("SYSTEM1_MODEL", "onejev-0.8b") +PORT = int(os.environ.get("SYSTEM1_PORT", "8017")) +IDLE = int(os.environ.get("SYSTEM1_IDLE_S", "600")) +last = [None] # idle clock starts once the server is up; a slow cold start must not be reaped + +_create_app = server.create_app + + +def create_app(*args, **kwargs): + app = _create_app(*args, **kwargs) + + @app.middleware("http") + async def touch(request, call_next): + last[0] = time.time() + try: + return await call_next(request) + finally: + last[0] = time.time() + + @app.on_event("startup") + async def announce(): + last[0] = time.time() + tmp = STATE + ".tmp" + with open(tmp, "w") as f: + json.dump({"url": f"http://127.0.0.1:{PORT}", "model": MODEL, "pid": os.getpid(), + "startedAt": time.strftime("%Y-%m-%dT%H:%M:%S")}, f) + os.replace(tmp, STATE) + + return app + + +server.create_app = create_app # qev.cli imports create_app at call time, so the patch takes + + +def reaper(): + while True: + time.sleep(15) + if last[0] is not None and time.time() - last[0] > IDLE: + try: + with open(STATE) as f: + if json.load(f).get("pid") == os.getpid(): + os.unlink(STATE) + except Exception: + pass + os._exit(0) + + +def main(): + import torch + + device = os.environ.get("SYSTEM1_DEVICE") or ("mps" if torch.backends.mps.is_available() else "cpu") + threading.Thread(target=reaper, daemon=True).start() + sys.argv = ["qev", "serve", "--model", MODEL, "--name", MODEL, "--device", device, "--dtype", "bfloat16", + "--head-dtype", "float32", "--fork-mode", "sequential", "--no-cuda-graphs", "--no-gpu-preprocess", + "--host", "127.0.0.1", "--port", str(PORT)] + from qev.cli import main as qev_main + + qev_main() + + +if __name__ == "__main__": + main() diff --git a/packages/system1/set-key.py b/packages/system1/set-key.py new file mode 100644 index 0000000..9c39747 --- /dev/null +++ b/packages/system1/set-key.py @@ -0,0 +1,41 @@ +#!/usr/bin/env python3 +"""Save the user's OpenRouter key for hosted System 1 (TypeSafe Jev) without it touching chat or the screen. + +Hidden prompt → ~/.relay/tool-secrets.json {"system1": {"OPENROUTER_API_KEY": ...}} at 0600, merged with the +existing tool secrets. Prints nothing about the key itself. +""" +import getpass +import json +import os +import urllib.error +import urllib.request + +path = os.path.join(os.environ.get("RELAY_DIR", os.path.expanduser("~/.relay")), "tool-secrets.json") +key = getpass.getpass("Paste your OpenRouter key (hidden), then Enter: ").strip() +if not key.startswith("sk-or-"): + raise SystemExit("That doesn't look like an OpenRouter key (they start with sk-or-). Nothing saved.") +# Hidden input makes a double paste invisible; collapse an exact repeat. +half = len(key) // 2 +if len(key) % 2 == 0 and key[:half] == key[half:]: + key = key[:half] + print("You pasted it twice; kept one copy.") +# Check it with OpenRouter before saving, so a bad key never lands silently. +try: + urllib.request.urlopen(urllib.request.Request("https://openrouter.ai/api/v1/key", + headers={"Authorization": "Bearer " + key}), timeout=15) +except urllib.error.HTTPError as e: + raise SystemExit(f"OpenRouter rejected that key ({e.code}). Nothing saved.") +except OSError as e: + raise SystemExit(f"Couldn't reach OpenRouter to check the key ({e}). Nothing saved.") +try: + with open(path) as f: + data = json.load(f) +except FileNotFoundError: + data = {} +data.setdefault("system1", {})["OPENROUTER_API_KEY"] = key +fd = os.open(path + ".tmp", os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600) +with os.fdopen(fd, "w") as f: + json.dump(data, f) +os.replace(path + ".tmp", path) +os.chmod(path, 0o600) +print("Saved. System 1 will use hosted Jev now.") diff --git a/plugin/skills/system1/SKILL.md b/plugin/skills/system1/SKILL.md new file mode 100644 index 0000000..19e7cc6 --- /dev/null +++ b/plugin/skills/system1/SKILL.md @@ -0,0 +1,64 @@ +--- +name: system1 +description: Fast, typed judgment ("System 1 thinking") via Switchboard's switchboard_decide tool — a System One model (hosted TypeSafe Jev via OpenRouter, or an opt-in local engine) that reads a state (text, objects, screenshots) and answers typed questions with calibrated probabilities instead of generated text. Use when a judgment must be made over MANY items (triage hundreds of grep hits, files, log lines, test failures, screenshots, candidate branches × scenarios) before spending your own context on them, or for a cheap yes/no gate. Triggers - "/system1", "system 1 this", "triage these fast", "score all of these", "classify these", "filter the grep results". +--- + +# System 1 + +You are System 2: slow, careful, expensive context. `switchboard_decide` is System 1: a +decision model that answers typed questions in one forward pass, with no generated text. Use it to **decide what deserves +your attention**, never as the final word on anything that matters. + +## When it pays off + +- **Volume.** 50+ items that each need the same judgment: grep hits ("does this line implement X?"), + files ("is this relevant to the bug?"), log lines ("error that matters?"), screenshots + ("does this screen show the bug?"), branch × scenario grids. +- **Out of the loop.** Hooks and routines where no LLM is running (e.g. a Stop hook asking "does this message need a decision?"). + +For a handful of items, just read them yourself. The tool call costs more than it saves. + +## Calling it + +```json +{ + "state": "src/auth/session.ts:88: if (token.exp < now) return refresh(token)", + "questions": { + "relevant": {"type": "noul", "instructions": "This line decides whether a session token has expired."}, + "kind": {"type": "choice", "instructions": "What does this line do?", + "criteria": {"check": "checks a condition", "call": "calls another function", "data": "defines data"}}, + "risk": {"type": "score", "instructions": "How risky is changing this line?", "criteria": ["low", "medium", "high"]} + } +} +``` + +- `noul` → `{"noul": p}` (probability of yes). `choice` → `{"choice", "confidence", "probabilities"}`. + `score` → `{"score", "confidence", "probabilities"}` over the ordered levels. +- Images: put `` in the state and pass `media: [{"type":"image","path":"/abs/shot.png"}]`. +- One call = one state. For many items, call once per item (the engine caches the questions), or pack a + short list into one state and ask per-item `noul` questions. + +## Recall: jevgrep over what the user saw + +`switchboard_recall` searches the local screen journal (`~/.relay/journal`, written by the Switchboard app +while `~/.relay/journal-on` exists) by meaning: a local keyword prefilter, then System 1 scores each +candidate line. Use it for "what did X say about Y", "where did I see Z", "what was I doing Tuesday". +Phrase the query with words likely to be on screen (names, project, topic), answer from the matches, and +cite time + app. `switchboard_journal` reports status and turns recording on/off or forgets the last N +minutes — only when the user asks. + +## Rules + +1. **Filter, then read.** Use it to rank/cut a list, then read the survivors yourself before acting. +2. **Respect uncertainty.** `noul` between 0.3 and 0.7, or `confidence` < 0.7 → treat as unknown, not as an answer. +3. **Never gate anything irreversible on it** (deletes, sends, consent). The user decides those at the notch. +4. **Say when you used it**, e.g. "System 1 cut 412 hits to 23; I read those." +5. **Know where the state goes.** The result says which `engine` answered. `hosted` = TypeSafe Jev via + OpenRouter: the state leaves this Mac, so never send secrets, keys, credentials or private user data, + and send code only when the user is fine with that. Hosted takes text/objects only (no images yet). +6. **No engine?** Tell the user; don't set one up silently. Hosted: they add their own OpenRouter key as + `OPENROUTER_API_KEY` or in `~/.relay/tool-secrets.json` under `"system1"`. Local (`packages/system1/install.sh`, + OneJev) only makes sense on an NVIDIA box; on Apple Silicon it measured 30-45s per decision. + +The engine speaks TypeSafe's `/v1/systemone` contract, so `SYSTEM1_URL` can point at any compatible +server (Kev, Von, a GPU box running OneJev) without changing callers. From 49555b085045d8d6d2c9dcbd7f020dc11bf18860 Mon Sep 17 00:00:00 2001 From: sameeeeeeep <191771413+sameeeeeeep@users.noreply.github.com> Date: Tue, 29 Sep 2026 14:33:44 +0530 Subject: [PATCH 2/4] Switchboard 0.4.0 Co-Authored-By: Claude Opus 5.5 --- packages/menubar/Info.plist | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/packages/menubar/Info.plist b/packages/menubar/Info.plist index 3df62a1..72890c0 100644 --- a/packages/menubar/Info.plist +++ b/packages/menubar/Info.plist @@ -13,7 +13,7 @@ CFBundlePackageType APPL CFBundleShortVersionString - 0.3.20 + 0.4.0 CFBundleURLTypes @@ -26,7 +26,7 @@ CFBundleVersion - 41 + 42 LSMinimumSystemVersion 13.0 LSUIElement From c14301bf2ef91a65a6634d20dd235cd92734f220 Mon Sep 17 00:00:00 2001 From: sameeeeeeep <191771413+sameeeeeeep@users.noreply.github.com> Date: Tue, 29 Sep 2026 18:27:34 +0530 Subject: [PATCH 3/4] Switchboard 0.4.1: Journal tab, meeting transcripts, per-sense switches - Journal tab in the OS panel (Knowledge): a day strip with a calls lane, sessions (one session per meeting), calls as Me/Them conversations, local search, app filters, forget a session or the last hour. - Meeting audio (CallAudio.swift): a background watch spots Meet/Zoom/Teams/FaceTime and asks at the notch before transcribing. Mic = Me, the call app's audio = Them, transcribed on-device by whisper; text only, audio never kept; stops when the meeting closes. Mic falls back if echo cancellation returns silence; echoed "Me" lines are dropped. - Named speakers from the meeting's own captions (CallCaptions.swift). - Screen journal: a last look at the window being left, 20 s reads of call windows. - Senses (Senses.swift): screen, meeting audio and camera each switched on their own, on notch hover, on the loose pill and in the Journal tab. - Eyes follow the pointer; lime = screen, amber = listening, coral = camera. - Camera presence + people tagging (CameraPresence/CameraPanel.swift), parked: hidden unless "cameraLab": true. Co-Authored-By: Claude Opus 5.5 --- packages/menubar/CallAudio.swift | 365 ++++++++++++++++ packages/menubar/CallCaptions.swift | 170 ++++++++ packages/menubar/CameraPanel.swift | 211 ++++++++++ packages/menubar/CameraPresence.swift | 363 ++++++++++++++++ packages/menubar/Info.plist | 6 +- packages/menubar/LooseNotch.swift | 14 +- packages/menubar/OSShellView.swift | 11 +- packages/menubar/OSSurfaceJournal.swift | 530 ++++++++++++++++++++++++ packages/menubar/Relay.entitlements | 3 + packages/menubar/RelayMenuBar.swift | 44 +- packages/menubar/ScreenJournal.swift | 88 +++- packages/menubar/Senses.swift | 102 +++++ packages/menubar/build.sh | 2 +- 13 files changed, 1867 insertions(+), 42 deletions(-) create mode 100644 packages/menubar/CallAudio.swift create mode 100644 packages/menubar/CallCaptions.swift create mode 100644 packages/menubar/CameraPanel.swift create mode 100644 packages/menubar/CameraPresence.swift create mode 100644 packages/menubar/OSSurfaceJournal.swift create mode 100644 packages/menubar/Senses.swift diff --git a/packages/menubar/CallAudio.swift b/packages/menubar/CallAudio.swift new file mode 100644 index 0000000..6da68d8 --- /dev/null +++ b/packages/menubar/CallAudio.swift @@ -0,0 +1,365 @@ +// CallAudio — Granola-style call transcripts into the screen journal, entirely on this Mac. +// +// Two streams give "Me" and "Them" without any speaker model: the MICROPHONE is the user (macOS voice +// processing on, so the other side's voices from the speakers are echo-cancelled out of it), and the CALL +// APP'S OWN AUDIO (ScreenCaptureKit, filtered to that one app) is everyone else. Every 20 s each stream that +// actually has speech is written to a temporary WAV, transcribed by whisper.cpp, and deleted; the text lands +// in the journal as "Me: …" / "Them: …" lines tagged call + source audio, so recall, memory and cards work on it. +// +// CONSENT: a background watch scans window names every 10 s for a meeting (Meet code, Zoom / Teams meeting, +// FaceTime), whether or not it is in front, and ASKS at the notch before transcribing it ("audio": "ask", or true). +// "audio": "always" skips the question; false turns it off. One answer per meeting. Audio is never kept, only +// text; it stops within ~20 s of the meeting window closing. While it listens, the notch eyes turn amber. +import AppKit +import AVFoundation +import ScreenCaptureKit + +final class CallAudio: NSObject, SCStreamOutput, SCStreamDelegate { + static let shared = CallAudio() + + /// Main-thread callback: true while listening (drives the amber eyes). + var onListening: ((Bool) -> Void)? + /// Writes one journal entry (ScreenJournal.record); set at wiring time. + var record: (([String: Any]) -> Void)? + + private let q = DispatchQueue(label: "switchboard.call-audio", qos: .utility) + private var stream: SCStream? + private var engine: AVAudioEngine? + private var converter: AVAudioConverter? + private var them: [Float] = [], me: [Float] = [] + private var timer: DispatchSourceTimer? + private var callPid: pid_t = 0 + private var callBundle = "", callApp = "", callTitle = "" + private(set) var active = false + private var starting = false + private let rate: Double = 16_000 + // meeting key → "transcribe" | "speakers" | "skip"; kept in ~/.relay/call-audio.json so a restart doesn't re-ask + private lazy var decided: [String: String] = { + guard let d = FileManager.default.contents(atPath: decisionsPath), + let j = try? JSONSerialization.jsonObject(with: d) as? [String: String] else { return [:] } + return j + }() { didSet { + if let d = try? JSONSerialization.data(withJSONObject: decided) { + FileManager.default.createFile(atPath: decisionsPath, contents: d, attributes: [.posixPermissions: 0o600]) + } + } } + private var decisionsPath: String { (NSHomeDirectory() as NSString).appendingPathComponent(".relay/call-audio.json") } + private var asking: (key: String, runId: String, pid: pid_t, app: String, bundle: String, title: String)? + private var callKey = "" + private var watchTimer: DispatchSourceTimer? + private var voiceProcessing = true + private var silentMicChunks = 0 + private var relay: String { (NSHomeDirectory() as NSString).appendingPathComponent(".relay") } + + // MARK: lifecycle + + /// Start the background meeting watch (once, at launch). Cheap: one window-list read every 10 s. + func arm() { + q.async { + guard self.watchTimer == nil else { return } + let t = DispatchSource.makeTimerSource(queue: self.q) + t.schedule(deadline: .now() + 5, repeating: 10) + t.setEventHandler { [weak self] in self?.watch() } + t.resume() + self.watchTimer = t + } + } + + /// "ask" (default when true), "always", or "off" — from ~/.relay/journal.json "audio". + private func mode() -> String { + // Independent of the screen journal: meetings can be transcribed with the screen sense off. + guard let d = FileManager.default.contents(atPath: (relay as NSString).appendingPathComponent("journal.json")), + let j = try? JSONSerialization.jsonObject(with: d) as? [String: Any] else { return "off" } + if let b = j["audio"] as? Bool { return b ? "ask" : "off" } + if let s = j["audio"] as? String, ["ask", "always"].contains(s) { return s } + return "off" + } + + private func watch() { + let m = mode() + if m == "off" { if active { stop(reason: "audio off") }; asking = nil; return } + if let a = asking { readAnswer(a); return } + guard !active, !starting, let meeting = findMeeting() else { return } + switch decided[meeting.key] ?? (m == "always" ? "transcribe" : nil) { + case "transcribe"?, "speakers"?: begin(meeting) + case "skip"?: return + default: ask(meeting) + } + } + + private typealias Meeting = (key: String, pid: pid_t, app: String, bundle: String, title: String) + + /// Any window (front or not) that is a meeting: a Meet code in the name, a Zoom / Teams meeting window, FaceTime. + private func findMeeting() -> Meeting? { + guard let infos = CGWindowListCopyWindowInfo([.optionAll, .excludeDesktopElements], kCGNullWindowID) as? [[String: Any]] else { return nil } + for info in infos { + guard let name = info[kCGWindowName as String] as? String, !name.isEmpty, + let pid = info[kCGWindowOwnerPID as String] as? pid_t, + (info[kCGWindowLayer as String] as? Int) == 0, + let app = NSRunningApplication(processIdentifier: pid) else { continue } + let bundle = (app.bundleIdentifier ?? "").lowercased(), n = name.lowercased() + var key: String? + if let r = n.range(of: #"[a-z]{3}-[a-z]{4}-[a-z]{3}"#, options: .regularExpression), n.contains("meet") { key = "meet:" + n[r] } + else if bundle == "us.zoom.xos", n.contains("meeting") { key = "zoom:" + n } + else if bundle.hasPrefix("com.microsoft.teams"), n.contains("meeting") || n.contains("call") { key = "teams:" + n } + else if bundle == "com.apple.facetime", !n.isEmpty, n != "facetime" { key = "facetime:" + n } + if let key { return (key, pid, app.localizedName ?? bundle, bundle, name) } + } + return nil + } + + private func begin(_ m: Meeting) { + starting = true + callKey = m.key; callPid = m.pid; callBundle = m.bundle; callApp = m.app; callTitle = m.title + Task { await self.start() } + } + + /// One notch card per meeting. Uses the app's own guide-run.json protocol; skipped if another card is queued. + private func ask(_ m: Meeting) { + let runPath = (relay as NSString).appendingPathComponent("guide-run.json") + guard !FileManager.default.fileExists(atPath: runPath) else { return } + let runId = UUID().uuidString.lowercased() + let label = m.key.hasPrefix("meet:") ? "Meet " + m.key.dropFirst(5) : m.title + let run: [String: Any] = [ + "runId": runId, "title": "Meeting: \(label)", "source": "Switchboard · call audio", "sourceId": "call-audio", + "mode": "teach", + "steps": [[ + "id": "transcribe", "placement": "notch", + "text": "Transcribe this meeting? Text only, on this Mac. Audio is never kept.", + "say": "You're in a meeting. Want me to transcribe it?", + "options": [ + ["id": "transcribe", "label": "Transcribe", "detail": "Me and Them, into your journal", "recommended": true], + ["id": "speakers", "label": "Transcribe + identify speakers", "detail": "Names each person on the other side (coming soon: Me / Them for now)"], + ["id": "skip", "label": "Not this meeting", "detail": "Won't ask again for this one"], + ], + ]], + ] + guard let data = try? JSONSerialization.data(withJSONObject: run) else { return } + let tmp = runPath + ".\(runId).tmp" + FileManager.default.createFile(atPath: tmp, contents: data, attributes: [.posixPermissions: 0o600]) + do { try FileManager.default.linkItem(atPath: tmp, toPath: runPath) } catch { try? FileManager.default.removeItem(atPath: tmp); return } + try? FileManager.default.removeItem(atPath: tmp) + asking = (m.key, runId, m.pid, m.app, m.bundle, m.title) + godLog("call audio: asked about \(m.key)") + } + + private func readAnswer(_ a: (key: String, runId: String, pid: pid_t, app: String, bundle: String, title: String)) { + let p = (relay as NSString).appendingPathComponent("guide-results/\(a.runId).json") + guard let d = FileManager.default.contents(atPath: p), + let j = try? JSONSerialization.jsonObject(with: d) as? [String: Any] else { + if findMeeting()?.key != a.key { asking = nil } // meeting gone before an answer: drop the question + return + } + let step = (j["results"] as? [[String: Any]])?.first + let note = (step?["feedback"] as? [String: Any])?["note"] as? String + let pick = note == nil ? (step?["chosenOption"] as? String ?? "skip") : "skip" // a typed note is not a yes + decided[a.key] = pick + asking = nil + godLog("call audio: \(a.key) → \(pick)") + if pick != "skip" { begin((a.key, a.pid, a.app, a.bundle, a.title)) } + } + + func stopIfActive(reason: String) { q.async { if self.active { self.stop(reason: reason) } } } + + private func start() async { + do { + let content = try await SCShareableContent.excludingDesktopWindows(false, onScreenWindowsOnly: false) + guard let display = content.displays.first, + let scApp = content.applications.first(where: { $0.processID == callPid }) else { + q.async { self.starting = false }; godLog("call audio: call app not shareable"); return + } + let cfg = SCStreamConfiguration() + cfg.capturesAudio = true + cfg.sampleRate = Int(rate) + cfg.channelCount = 1 + cfg.excludesCurrentProcessAudio = true + cfg.width = 2; cfg.height = 2 // audio is the point; keep video negligible + cfg.minimumFrameInterval = CMTime(value: 1, timescale: 1) + let s = SCStream(filter: SCContentFilter(display: display, including: [scApp], exceptingWindows: []), + configuration: cfg, delegate: self) + try s.addStreamOutput(self, type: .audio, sampleHandlerQueue: q) + try s.addStreamOutput(self, type: .screen, sampleHandlerQueue: q) // no-op sink so frames aren't logged as dropped + try await s.startCapture() + q.async { + self.stream = s + self.startMic() + self.active = true; self.starting = false + self.startTimer() + DispatchQueue.main.async { self.onListening?(true) } + // Named speakers from the meeting's own captions (CallCaptions.swift), alongside the audio. + CallCaptions.shared.start(pid: self.callPid, meetingKey: self.callKey, app: self.callApp, bundle: self.callBundle, + title: self.callTitle, record: { self.record?($0) }) + godLog("call audio ON: \(self.callApp) '\(self.callTitle.prefix(40))'") + } + } catch { + q.async { self.starting = false } + godLog("call audio: could not start (\(error.localizedDescription))") + } + } + + private func startMic() { + let eng = AVAudioEngine() + let input = eng.inputNode + try? input.setVoiceProcessingEnabled(voiceProcessing) // echo-cancel the far side out of "Me" + if #available(macOS 14.0, *) { + input.voiceProcessingOtherAudioDuckingConfiguration = .init(enableAdvancedDucking: false, duckingLevel: .min) + } + let inFmt = input.outputFormat(forBus: 0) + guard let outFmt = AVAudioFormat(commonFormat: .pcmFormatFloat32, sampleRate: rate, channels: 1, interleaved: false), + let conv = AVAudioConverter(from: inFmt, to: outFmt) else { godLog("call audio: mic format unsupported"); return } + converter = conv + input.installTap(onBus: 0, bufferSize: 4096, format: inFmt) { [weak self] buf, _ in + guard let self, let conv = self.converter else { return } + let cap = AVAudioFrameCount(Double(buf.frameLength) * self.rate / inFmt.sampleRate) + 64 + guard let out = AVAudioPCMBuffer(pcmFormat: outFmt, frameCapacity: cap) else { return } + var fed = false + conv.convert(to: out, error: nil) { _, status in + if fed { status.pointee = .noDataNow; return nil } + fed = true; status.pointee = .haveData; return buf + } + guard let ch = out.floatChannelData?[0] else { return } + let samples = Array(UnsafeBufferPointer(start: ch, count: Int(out.frameLength))) + self.q.async { self.me.append(contentsOf: samples) } + } + do { try eng.start(); engine = eng } catch { godLog("call audio: mic unavailable (\(error.localizedDescription))") } + } + + private func startTimer() { + let t = DispatchSource.makeTimerSource(queue: q) + t.schedule(deadline: .now() + 20, repeating: 20) + t.setEventHandler { [weak self] in + guard let self else { return } + if !self.callWindowOpen() { self.stop(reason: "call ended"); return } + self.flush() + } + t.resume() + timer = t + } + + private func stop(reason: String) { + flush() + timer?.cancel(); timer = nil + if let s = stream { Task { try? await s.stopCapture() } } + stream = nil + engine?.inputNode.removeTap(onBus: 0); engine?.stop(); engine = nil; converter = nil + them.removeAll(); me.removeAll() + CallCaptions.shared.stop() + active = false + DispatchQueue.main.async { self.onListening?(false) } + godLog("call audio OFF (\(reason))") + } + + /// The call is still on while its app has an on-screen window whose title looks like the call's. + private func callWindowOpen() -> Bool { + guard let infos = CGWindowListCopyWindowInfo([.optionAll], kCGNullWindowID) as? [[String: Any]] else { return false } + // Window-server names differ from AX titles ("Meet – abc-defg-hij 🔊" vs "Meet – abc-defg-hij – Microphone…"), + // so match on the Meet code when there is one, else on the title's first words. + let t = callTitle.lowercased() + let key = t.range(of: #"[a-z]{3}-[a-z]{4}-[a-z]{3}"#, options: .regularExpression).map { String(t[$0]) } + ?? String(t.prefix(12)) + if findMeeting()?.key == callKey { return true } + return infos.contains { info in + (info[kCGWindowOwnerPID as String] as? pid_t) == callPid && + ((info[kCGWindowName as String] as? String)?.lowercased().contains(key) ?? false) + } + } + + // MARK: capture + + func stream(_ stream: SCStream, didOutputSampleBuffer sb: CMSampleBuffer, of type: SCStreamOutputType) { + guard type == .audio else { return } + try? sb.withAudioBufferList { abl, _ in + guard let first = abl.first, let data = first.mData else { return } + let n = Int(first.mDataByteSize) / MemoryLayout.size + them.append(contentsOf: UnsafeBufferPointer(start: data.assumingMemoryBound(to: Float.self), count: n)) + } + } + func stream(_ stream: SCStream, didStopWithError error: Error) { + q.async { if self.active { self.stop(reason: "stream stopped: \(error.localizedDescription)") } } + } + + // MARK: transcription + + private func flush() { + let pairs = [("Them", them), ("Me", me)] + them.removeAll(keepingCapacity: true); me.removeAll(keepingCapacity: true) + var lines: [String] = [] + var heard: [String] = [] + // While captions are flowing they already name the other side; unnamed "Them" audio would duplicate them. + let captionsLive = Date().timeIntervalSince(CallCaptions.shared.lastCaption) < 60 + for (who, samples) in pairs where !(who == "Them" && captionsLive) { + let rms = samples.isEmpty ? 0 : sqrt(samples.reduce(0) { $0 + $1 * $1 } / Float(samples.count)) + // Voice processing can hand back pure zeros when the call app also holds the mic: fall back once. + if who == "Me", samples.count > Int(rate), rms == 0, voiceProcessing { + silentMicChunks += 1 + if silentMicChunks >= 1 { + voiceProcessing = false + engine?.inputNode.removeTap(onBus: 0); engine?.stop(); engine = nil; converter = nil + startMic() + godLog("call audio: mic was silent with echo cancellation; restarted without it") + } + } + heard.append("\(who) \(String(format: "%.1f", Double(samples.count) / rate))s rms \(String(format: "%.4f", rms))") + guard samples.count > Int(rate), rms > 0.004 else { continue } // silence: don't spend CPU on it + for s in transcribe(samples) { lines.append("\(who): \(s)") } + } + // Without echo cancellation the mic hears the speakers: drop "Me" lines that mostly repeat "Them". + let words = { (l: String) in Set(l.lowercased().split(whereSeparator: { !$0.isLetter }).map(String.init)) } + let themWords = lines.filter { $0.hasPrefix("Them:") }.reduce(into: Set()) { $0.formUnion(words($1)) } + lines.removeAll { l in + guard l.hasPrefix("Me:") else { return false } + let w = words(String(l.dropFirst(3))) + return w.count >= 5 && Double(w.intersection(themWords).count) / Double(w.count) > 0.7 + } + godLog("call audio chunk: \(heard.joined(separator: " · ")) → \(lines.count) lines") + guard !lines.isEmpty else { return } + record?(["t": ISO8601DateFormatter().string(from: Date()), "app": callApp, "bundle": callBundle, + "window": callTitle, "url": NSNull(), "via": "audio", "source": "audio", "call": true, "lines": lines]) + } + + private func transcribe(_ samples: [Float]) -> [String] { + guard let bin = Self.whisperBin(), let model = Self.whisperModel() else { godLog("call audio: whisper not found"); return [] } + let tmp = (NSHomeDirectory() as NSString).appendingPathComponent(".relay/tmp") + try? FileManager.default.createDirectory(atPath: tmp, withIntermediateDirectories: true, attributes: [.posixPermissions: 0o700]) + let wav = (tmp as NSString).appendingPathComponent("call-\(UUID().uuidString).wav") + defer { try? FileManager.default.removeItem(atPath: wav) } + guard Self.writeWAV(samples, rate: Int(rate), to: wav) else { return [] } + let p = Process(), out = Pipe() + p.executableURL = URL(fileURLWithPath: bin) + p.arguments = ["-m", model, "-f", wav, "-nt", "-np", "-l", "en"] + p.standardOutput = out; p.standardError = FileHandle.nullDevice + do { try p.run() } catch { return [] } + let data = out.fileHandleForReading.readDataToEndOfFile() + p.waitUntilExit() + let text = String(decoding: data, as: UTF8.self) + return text.split(whereSeparator: \.isNewline) + .map { $0.trimmingCharacters(in: .whitespaces) } + .filter { $0.count >= 3 && !$0.hasPrefix("[") && !$0.hasPrefix("(") } // drop [BLANK_AUDIO], (music)… + } + + static func whisperBin() -> String? { + let bundled = ((Bundle.main.resourcePath ?? "") as NSString).appendingPathComponent("stt/whisper-cli") + return [bundled, "/opt/homebrew/bin/whisper-cli", "/usr/local/bin/whisper-cli"].first { FileManager.default.fileExists(atPath: $0) } + } + /// Calls prefer the user's base.en model (clearer on real speech) over the bundled tiny one. + static func whisperModel() -> String? { + let dir = (NSHomeDirectory() as NSString).appendingPathComponent(".relay/models") + if let f = (try? FileManager.default.contentsOfDirectory(atPath: dir))?.first(where: { $0.hasSuffix(".bin") && $0.contains("base.en") }) { + return (dir as NSString).appendingPathComponent(f) + } + let bundled = ((Bundle.main.resourcePath ?? "") as NSString).appendingPathComponent("stt/ggml-tiny.en.bin") + return FileManager.default.fileExists(atPath: bundled) ? bundled : nil + } + + static func writeWAV(_ samples: [Float], rate: Int, to path: String) -> Bool { + var pcm = Data(capacity: samples.count * 2) + for s in samples { var v = Int16(max(-1, min(1, s)) * 32767).littleEndian; withUnsafeBytes(of: &v) { pcm.append(contentsOf: $0) } } + var h = Data() + func u32(_ v: UInt32) { var x = v.littleEndian; withUnsafeBytes(of: &x) { h.append(contentsOf: $0) } } + func u16(_ v: UInt16) { var x = v.littleEndian; withUnsafeBytes(of: &x) { h.append(contentsOf: $0) } } + h.append(contentsOf: Array("RIFF".utf8)); u32(UInt32(36 + pcm.count)); h.append(contentsOf: Array("WAVE".utf8)) + h.append(contentsOf: Array("fmt ".utf8)); u32(16); u16(1); u16(1); u32(UInt32(rate)); u32(UInt32(rate * 2)); u16(2); u16(16) + h.append(contentsOf: Array("data".utf8)); u32(UInt32(pcm.count)) + return FileManager.default.createFile(atPath: path, contents: h + pcm, attributes: [.posixPermissions: 0o600]) + } +} diff --git a/packages/menubar/CallCaptions.swift b/packages/menubar/CallCaptions.swift new file mode 100644 index 0000000..9577a48 --- /dev/null +++ b/packages/menubar/CallCaptions.swift @@ -0,0 +1,170 @@ +// CallCaptions — named speakers for call transcripts, read from the meeting's own live captions. +// +// Meet / Zoom / Teams already know who is talking: with captions on, they draw "Name" + what that person said at +// the bottom of the call window. While CallAudio is transcribing a meeting (the user said yes at the notch), this +// reads ONLY the bottom third of that window every 3 s (screencapture -l works even when the call is behind other +// windows), runs fast on-device OCR, splits it into speaker blocks, and journals each COMPLETE sentence once as +// "Name: sentence" ("You" → "Me"). Captions scroll away within seconds, which is why it polls this often and +// this small. Nothing is stored except the text lines; the crop is deleted immediately. +import AppKit +import Vision + +final class CallCaptions { + static let shared = CallCaptions() + + private let q = DispatchQueue(label: "switchboard.call-captions", qos: .utility) + private var timer: DispatchSourceTimer? + private var pid: pid_t = 0 + private var meetingKey = "" + private var app = "", bundle = "", title = "" + private var record: (([String: Any]) -> Void)? + private var seen: [String: Date] = [:] // "speaker|sentence" → when journaled + private(set) var lastCaption = Date.distantPast // CallAudio drops unnamed "Them" lines while captions flow + private var known: Set = [] // participant names, learned from the journal's tile captures + private var knownAt = Date.distantPast + + func start(pid: pid_t, meetingKey: String, app: String, bundle: String, title: String, record: @escaping ([String: Any]) -> Void) { + q.async { + self.stopLocked() + self.pid = pid; self.meetingKey = meetingKey; self.app = app; self.bundle = bundle; self.title = title + self.record = record + let t = DispatchSource.makeTimerSource(queue: self.q) + t.schedule(deadline: .now() + 2, repeating: 3) + t.setEventHandler { [weak self] in self?.tick() } + t.resume() + self.timer = t + } + } + + func stop() { q.async { self.stopLocked() } } + private func stopLocked() { timer?.cancel(); timer = nil; seen.removeAll() } + + private func tick() { + guard let wid = windowID(), let lines = ocrCaptionBand(wid), !lines.isEmpty else { return } + if Date().timeIntervalSince(knownAt) > 60 { known = learnNames(); knownAt = Date() } + let blocks = Self.parse(lines, known: known) + var out: [String] = [] + let now = Date() + for b in blocks { + // only COMPLETE sentences; the last one in a live block may still be growing + for s in Self.sentences(b.text) where s.complete { + let key = b.speaker.lowercased() + "|" + s.text.lowercased() + if seen[key] != nil { continue } + seen[key] = now + out.append("\(b.speaker): \(s.text)") + } + } + if seen.count > 4000 { seen = seen.filter { now.timeIntervalSince($0.value) < 1800 } } + guard !out.isEmpty else { return } + lastCaption = now + record?(["t": ISO8601DateFormatter().string(from: now), "app": app, "bundle": bundle, "window": title, + "url": NSNull(), "via": "captions", "source": "audio", "call": true, "lines": out]) + } + + /// The meeting's window (front or not): same pid, name containing the meeting code / title stem. + private func windowID() -> Int? { + guard let infos = CGWindowListCopyWindowInfo([.optionAll, .excludeDesktopElements], kCGNullWindowID) as? [[String: Any]] else { return nil } + let stem = meetingKey.split(separator: ":").last.map(String.init) ?? "" + return infos.first { i in + (i[kCGWindowOwnerPID as String] as? pid_t) == pid && (i[kCGWindowLayer as String] as? Int) == 0 && + ((i[kCGWindowName as String] as? String)?.lowercased().contains(stem) ?? false) + }?[kCGWindowNumber as String] as? Int + } + + /// Capture the window, keep the bottom third (where captions live), OCR it fast, top-to-bottom lines. + private func ocrCaptionBand(_ wid: Int) -> [String]? { + let tmp = (NSHomeDirectory() as NSString).appendingPathComponent(".relay/tmp") + try? FileManager.default.createDirectory(atPath: tmp, withIntermediateDirectories: true, attributes: [.posixPermissions: 0o700]) + let path = (tmp as NSString).appendingPathComponent("captions-\(UUID().uuidString).png") + defer { try? FileManager.default.removeItem(atPath: path) } + let cap = Process() + cap.executableURL = URL(fileURLWithPath: "/usr/sbin/screencapture") + cap.arguments = ["-x", "-o", "-l", String(wid), "-t", "png", path] + do { try cap.run(); cap.waitUntilExit() } catch { return nil } + guard let full = NSImage(contentsOfFile: path)?.cgImage(forProposedRect: nil, context: nil, hints: nil) else { return nil } + let bandH = full.height / 3 + guard let band = full.cropping(to: CGRect(x: 0, y: full.height - bandH, width: full.width, height: bandH)) else { return nil } + var out: [(y: CGFloat, text: String)] = [] + let req = VNRecognizeTextRequest { r, _ in + for o in (r.results as? [VNRecognizedTextObservation]) ?? [] { + if let t = o.topCandidates(1).first?.string.trimmingCharacters(in: .whitespaces), !t.isEmpty { + out.append((o.boundingBox.midY, t)) + } + } + } + req.recognitionLevel = .fast + req.usesLanguageCorrection = true + try? VNImageRequestHandler(cgImage: band, options: [:]).perform([req]) + return out.sorted { $0.y > $1.y }.map(\.text) // Vision's y grows upward + } + + /// Participant tiles repeat in every 20 s screen read of the call, so short name-shaped lines that recur ≥ 3 + /// times in today's journal entries for this meeting are its participants. + private func learnNames() -> Set { + let day = String(ISO8601DateFormatter().string(from: Date()).prefix(10)) + let path = (NSHomeDirectory() as NSString).appendingPathComponent(".relay/journal/\(day).jsonl") + guard let text = try? String(contentsOfFile: path, encoding: .utf8) else { return [] } + let stem = (meetingKey.split(separator: ":").last.map(String.init) ?? "").lowercased() + var counts: [String: Int] = [:] + for row in text.split(separator: "\n") { + guard let j = try? JSONSerialization.jsonObject(with: Data(row.utf8)) as? [String: Any], + (j["window"] as? String)?.lowercased().contains(stem) == true, (j["via"] as? String) != "captions", + (j["source"] as? String) != "audio", let lines = j["lines"] as? [String] else { continue } + for l in Set(lines) where Self.nameShaped(l) { counts[l.lowercased(), default: 0] += 1 } + } + return Set(counts.filter { $0.value >= 3 }.keys) + } + + static func nameShaped(_ l: String) -> Bool { + let words = l.split(separator: " ") + return (1...4).contains(words.count) && l.count <= 32 && words.allSatisfy { $0.allSatisfy { $0.isLetter } } + } + + // MARK: parsing (pure, testable) + + struct Block { var speaker: String; var text: String } + + /// A short line that looks like a name, followed by caption text, starts a speaker block. Call-control labels + /// ("Turn off captions", "Present now", clock times) are dropped. + static func parse(_ lines: [String], known: Set = []) -> [Block] { + let chrome = ["captions", "present", "raise hand", "leave call", "more options", "turn on", "turn off", "meeting details", + "chat", "activities", "host controls", "mic", "camera", "|"] + var blocks: [Block] = [] + for raw in lines { + let l = raw.trimmingCharacters(in: .whitespaces) + let low = l.lowercased() + if l.count < 2 || chrome.contains(where: { low.contains($0) }) || l.range(of: #"^\d{1,2}:\d{2}"#, options: .regularExpression) != nil { continue } + if looksLikeName(l, known: known) { + blocks.append(Block(speaker: l == "You" ? "Me" : l, text: "")) + } else if !blocks.isEmpty { + blocks[blocks.count - 1].text += (blocks[blocks.count - 1].text.isEmpty ? "" : " ") + l + } + } + return blocks.filter { !$0.text.isEmpty } + } + + /// "You", a known participant, or, before any are known, 2–4 words that all start with a capital. + static func looksLikeName(_ l: String, known: Set = []) -> Bool { + if l == "You" { return true } + guard nameShaped(l) else { return false } + if !known.isEmpty { return known.contains(l.lowercased()) } + let words = l.split(separator: " ") + return (2...4).contains(words.count) && words.allSatisfy { $0.first?.isUppercase ?? false } + } + + static func sentences(_ text: String) -> [(text: String, complete: Bool)] { + var out: [(String, Bool)] = [] + var cur = "" + for ch in text { + cur.append(ch) + if ".?!".contains(ch) { + let s = cur.trimmingCharacters(in: .whitespaces) + if s.count > 3 { out.append((s, true)) } + cur = "" + } + } + let rest = cur.trimmingCharacters(in: .whitespaces) + if !rest.isEmpty { out.append((rest, false)) } + return out + } +} diff --git a/packages/menubar/CameraPanel.swift b/packages/menubar/CameraPanel.swift new file mode 100644 index 0000000..12e4054 --- /dev/null +++ b/packages/menubar/CameraPanel.swift @@ -0,0 +1,211 @@ +// CameraPanel — see what the camera sees and tag people. A small window with the live camera image, a box on every +// face, a coral name tag on enrolled people and a "Tag" button on everyone else. Tagging asks for the person's name +// and their consent, then captures six samples of THAT face (CameraPresence.enrol(face:)). Opening the panel keeps +// the camera on while it's open, even if camera presence is off; closing it hands the camera back. +import AppKit +import AVFoundation +import SwiftUI + +final class CameraPanelController: NSObject, NSWindowDelegate { + static let shared = CameraPanelController() + private var window: NSWindow? + + func show() { + if let w = window { w.makeKeyAndOrderFront(nil); NSApp.activate(ignoringOtherApps: true); return } + let w = NSWindow(contentRect: NSRect(x: 0, y: 0, width: 640, height: 520), + styleMask: [.titled, .closable, .resizable, .fullSizeContentView], backing: .buffered, defer: false) + w.title = "Camera · people" + w.titlebarAppearsTransparent = true + w.backgroundColor = .black + w.isReleasedWhenClosed = false + w.minSize = NSSize(width: 480, height: 420) + w.contentView = NSHostingView(rootView: CameraPanelView()) + w.center() + w.delegate = self + window = w + w.makeKeyAndOrderFront(nil) + NSApp.activate(ignoringOtherApps: true) + } + func windowWillClose(_ notification: Notification) { + CameraPresence.shared.viewerClosed() + CameraPresence.shared.onFaces = nil + window = nil + } +} + +/// Hosts the live preview. The camera session arrives AFTER the window opens (it starts on the camera queue), so the +/// preview layer is attached whenever a session shows up, and resized with the view on every layout. +private final class PreviewNSView: NSView { + private var preview: AVCaptureVideoPreviewLayer? + override init(frame: NSRect) { super.init(frame: frame); wantsLayer = true; layer?.backgroundColor = NSColor.black.cgColor } + required init?(coder: NSCoder) { fatalError() } + func attach(_ session: AVCaptureSession?) { + guard let session, preview?.session !== session else { return } + preview?.removeFromSuperlayer() + let l = AVCaptureVideoPreviewLayer(session: session) + l.videoGravity = .resizeAspect + l.frame = bounds + layer?.addSublayer(l) + preview = l + } + override func layout() { super.layout(); preview?.frame = bounds } +} + +private struct PreviewView: NSViewRepresentable { + let session: AVCaptureSession? + func makeNSView(context: Context) -> PreviewNSView { let v = PreviewNSView(); v.attach(session); return v } + func updateNSView(_ v: PreviewNSView, context: Context) { v.attach(session) } +} + +struct CameraPanelView: View { + @State private var session: AVCaptureSession? = nil + @State private var faces: [(box: CGRect, name: String?)] = [] + @State private var tagging: CGRect? = nil + @State private var name = "" + @State private var consent = false + @State private var progress: Int? = nil + @State private var message: String? = nil + @State private var aspect: CGFloat = 4.0 / 3.0 + @State private var fixing: (box: CGRect, wrong: String)? = nil // "Not Arsheen? This is…" + @State private var people: [(id: String, name: String, consentedAt: String)] = [] + @State private var fixTo: String = "" + @State private var prompt: String? = nil // the current enrolment step, shown over the camera + + var body: some View { + VStack(alignment: .leading, spacing: 10) { + HStack { + VStack(alignment: .leading, spacing: 4) { + Text("CAMERA").font(.splMono(10.5)).tracking(1.8).foregroundColor(.inkFaint) + Text("Who's in view").font(.hanken(20, .semibold)).foregroundColor(.ink) + } + Spacer() + SensesPill(compact: false) + } + GeometryReader { geo in + // the preview is aspect-fit inside the box; map Vision's normalized, bottom-left boxes into it + let fit = fitRect(container: geo.size, aspect: aspect) + ZStack(alignment: .topLeading) { + PreviewView(session: session).frame(width: geo.size.width, height: geo.size.height) + if let p = prompt, progress != nil { + VStack(spacing: 6) { + Text(p).font(.hanken(20, .semibold)).foregroundColor(.white) + HStack(spacing: 4) { + ForEach(0.. 0 { aspect = CGFloat(d.width) / CGFloat(d.height) } + } + } + } + } + + private func fix(_ fx: (box: CGRect, wrong: String)) { + if fixTo == "new" { tagging = fx.box; fixing = nil; return } // becomes a normal tag with consent + guard let p = people.first(where: { $0.id == fixTo }) else { return } + progress = 0 + CameraPresence.shared.correct(face: fx.box, personId: p.id, name: p.name, progress: { progress = $0 }, prompt: { prompt = $0 }) { r in + progress = nil; prompt = nil + switch r { + case .success(let who): message = "Fixed: that's \(who). Their profile learned from this face."; fixing = nil + case .failure(let e): message = e.localizedDescription + } + } + } + + private func save() { + guard let box = tagging else { return } + let n = name.trimmingCharacters(in: .whitespaces) + progress = 0 + CameraPresence.shared.enrol(name: n, face: box, progress: { progress = $0 }, prompt: { prompt = $0 }) { r in + progress = nil; prompt = nil + switch r { + case .success(let who): message = "Tagged \(who). They'll be recognised from now on."; tagging = nil; name = ""; consent = false + case .failure(let e): message = e.localizedDescription + } + } + } + + private func fitRect(container: CGSize, aspect: CGFloat) -> CGRect { + let w = min(container.width, container.height * aspect), h = w / aspect + return CGRect(x: (container.width - w) / 2, y: (container.height - h) / 2, width: w, height: h) + } + /// Vision boxes are normalized with a bottom-left origin (the preview layer shows the frame unmirrored). + private func viewRect(_ b: CGRect, in fit: CGRect) -> CGRect { + let x = b.minX, y = 1 - b.maxY + return CGRect(x: fit.minX + x * fit.width, y: fit.minY + y * fit.height, width: b.width * fit.width, height: b.height * fit.height) + } +} diff --git a/packages/menubar/CameraPresence.swift b/packages/menubar/CameraPresence.swift new file mode 100644 index 0000000..2d1072f --- /dev/null +++ b/packages/menubar/CameraPresence.swift @@ -0,0 +1,363 @@ +// CameraPresence — camera step 1: presence only, no identities. +// +// Opt-in: runs only with "camera": true in ~/.relay/journal.json AND the journal recording. Takes low-resolution +// frames, analyses about one per second with Apple's on-device face detection, and derives three facts: +// at desk / away, how many people are in view, and whether the nearest face is turned toward the screen. Only +// CHANGES are journaled ("Away", "At desk", "2 people in view"), so the day strip shows when you stepped away. +// While the camera is on, the notch eyes turn coral (and the Mac's own green camera light is on). +// +// STEP 2, ENROLMENT (opt-in per person): the user adds a person who is present and agrees to be recognised. Six face +// samples become Apple Vision feature prints (on-device, no model download, no licensing strings attached), saved +// with the consent time in ~/.relay/faces/.json (0600). While presence runs, each face in view is compared ONLY +// against enrolled people; a match adds their name to the journal line. Faces of people not enrolled are counted, +// never stored. No images are ever saved. +import AVFoundation +import Vision + +final class CameraPresence: NSObject, AVCaptureVideoDataOutputSampleBufferDelegate { + static let shared = CameraPresence() + + var onCamera: ((Bool) -> Void)? // main thread: coral eyes + var record: (([String: Any]) -> Void)? + + private let q = DispatchQueue(label: "switchboard.camera-presence", qos: .utility) + private var session: AVCaptureSession? + private var watch: DispatchSourceTimer? + private var lastFrame = Date.distantPast + private var faces = 0, facing = false, lastFaceSeen = Date.distantPast + private var reported: (present: Bool, people: Int, names: [String])? = nil + private var pending: (present: Bool, people: Int, names: [String], since: Date)? = nil + private var names: [String] = [] + + struct Person { let id: String; let name: String; let consentedAt: String; let prints: [VNFeaturePrintObservation]; let threshold: Float } + private var people: [Person] = [] + private var peopleLoaded = false + /// Guided multi-angle enrolment (Face ID style): each step only accepts frames where the head is really in that pose. + private struct Enrol { + var name: String + var samples: [VNFeaturePrintObservation] = [] + var poses: [String] = [] + var sideSign: Double = 0 + let temporary: Bool + let done: (Result) -> Void + let progress: (Int) -> Void + let prompt: (String) -> Void + } + static let plan: [(pose: String, count: Int, say: String)] = [ + ("straight", 3, "Look straight at the camera"), + ("side", 2, "Now turn your head slowly to one side"), + ("other", 2, "Now the other side"), + ("tilt", 1, "Tilt your chin up or down a little"), + ] + static var planTotal: Int { plan.reduce(0) { $0 + $1.count } } + private var enrolling: Enrol? + // The Camera panel (CameraPanel.swift): live boxes while it's open; it keeps the camera on even if presence is off. + var onFaces: (([(box: CGRect, name: String?)]) -> Void)? // main thread, normalized Vision boxes + private var viewers = 0 + private var enrolTarget: CGRect? = nil + private var enrolInto: String? = nil // correcting a tag: add samples to this existing person + private(set) var captureSession: AVCaptureSession? { get { session } set { session = newValue } } + /// No profile may be looser than this (a shaky enrolment once produced 1.23 and matched everyone). + static let maxThreshold: Float = 0.45 + private static let facesDir = (NSHomeDirectory() as NSString).appendingPathComponent(".relay/faces") + + /// Arm once at launch: a cheap 5 s check of the setting starts or stops the camera. + func arm() { + q.async { + guard self.watch == nil else { return } + let t = DispatchSource.makeTimerSource(queue: self.q) + t.schedule(deadline: .now() + 3, repeating: 5) + t.setEventHandler { [weak self] in self?.check() } + t.resume() + self.watch = t + } + } + + private func enabled() -> Bool { + let relay = (NSHomeDirectory() as NSString).appendingPathComponent(".relay") + // Independent of the screen journal (Senses.swift): the camera has its own switch. + guard let d = FileManager.default.contents(atPath: (relay as NSString).appendingPathComponent("journal.json")), + let j = try? JSONSerialization.jsonObject(with: d) as? [String: Any] else { return false } + return j["camera"] as? Bool ?? false + } + + private func check() { + if !peopleLoaded { loadPeople() } + if enrolling != nil { return } // an enrolment owns the camera for a few seconds + let want = enabled() || viewers > 0 + if want && session == nil { start() } else if !want && session != nil { stop() } + if session != nil { settle() } + } + + private func start() { + switch AVCaptureDevice.authorizationStatus(for: .video) { + case .notDetermined: + AVCaptureDevice.requestAccess(for: .video) { _ in } // macOS shows its own prompt; retried next check + return + case .authorized: break + default: godLog("camera presence: camera access denied in System Settings"); return + } + guard let device = AVCaptureDevice.default(for: .video), + let input = try? AVCaptureDeviceInput(device: device) else { godLog("camera presence: no camera"); return } + let s = AVCaptureSession() + s.sessionPreset = .low + guard s.canAddInput(input) else { return } + s.addInput(input) + let out = AVCaptureVideoDataOutput() + out.alwaysDiscardsLateVideoFrames = true + out.setSampleBufferDelegate(self, queue: q) + guard s.canAddOutput(out) else { return } + s.addOutput(out) + // Ask the camera for as few frames as it supports; we only look at ~1 per second anyway. + if (try? device.lockForConfiguration()) != nil { + if let r = device.activeFormat.videoSupportedFrameRateRanges.min(by: { $0.minFrameRate < $1.minFrameRate }) { + device.activeVideoMinFrameDuration = r.maxFrameDuration + device.activeVideoMaxFrameDuration = r.maxFrameDuration + } + device.unlockForConfiguration() + } + s.startRunning() + session = s + DispatchQueue.main.async { self.onCamera?(true) } + godLog("camera presence ON") + } + + private func stop() { + session?.stopRunning(); session = nil + reported = nil; pending = nil + DispatchQueue.main.async { self.onCamera?(false) } + godLog("camera presence OFF") + } + + func captureOutput(_ output: AVCaptureOutput, didOutput sb: CMSampleBuffer, from connection: AVCaptureConnection) { + let every: TimeInterval = enrolling != nil ? 0.5 : (viewers > 0 ? 0.33 : 1) + guard Date().timeIntervalSince(lastFrame) >= every, let px = CMSampleBufferGetImageBuffer(sb) else { return } + lastFrame = Date() + let handler = VNImageRequestHandler(cvPixelBuffer: px, options: [:]) + let req = VNDetectFaceRectanglesRequest() + req.revision = VNDetectFaceRectanglesRequestRevision3 // continuous yaw / pitch / roll + try? handler.perform([req]) + let obs = (req.results ?? []).filter { $0.confidence > 0.6 } + faces = obs.count + if let nearest = obs.max(by: { $0.boundingBox.width < $1.boundingBox.width }) { + lastFaceSeen = Date() + facing = abs(nearest.yaw?.doubleValue ?? 0) < 0.4 + if var e = enrolling { // enrolment: the tagged face (else the nearest), one sample per 0.5 s + let face = enrolTarget.flatMap { t in obs.min { hypot($0.boundingBox.midX - t.midX, $0.boundingBox.midY - t.midY) < hypot($1.boundingBox.midX - t.midX, $1.boundingBox.midY - t.midY) } } ?? nearest + enrolTarget = enrolTarget == nil ? nil : face.boundingBox // follow the face as it moves + publish(obs.map { ($0.boundingBox, $0 === face ? e.name : nil) }) + // which step are we on, and is the head actually in that pose? + var step = 0, before = 0 + while step < Self.plan.count, e.samples.count >= before + Self.plan[step].count { before += Self.plan[step].count; step += 1 } + if step >= Self.plan.count { finishEnrolment(); return } + let yaw = face.yaw?.doubleValue ?? 0, pitch = face.pitch?.doubleValue ?? 0 + let want = Self.plan[step].pose + let ok: Bool = { + switch want { + case "straight": return abs(yaw) < 0.2 && abs(pitch) < 0.25 + case "side": return abs(yaw) > 0.3 + case "other": return abs(yaw) > 0.3 && (yaw > 0) != (e.sideSign > 0) + default: return abs(pitch) > 0.2 + } + }() + if ok, let fp = Self.print(handler, face) { + if want == "side" && e.sideSign == 0 { e.sideSign = yaw } + e.samples.append(fp); e.poses.append(want) + let n = e.samples.count + DispatchQueue.main.async { e.progress(n) } + } + // tell them the NEXT thing to do (or the current one, if this frame didn't count) + var s2 = 0, b2 = 0 + while s2 < Self.plan.count, e.samples.count >= b2 + Self.plan[s2].count { b2 += Self.plan[s2].count; s2 += 1 } + let say = s2 < Self.plan.count ? Self.plan[s2].say : "Done" + DispatchQueue.main.async { e.prompt(say) } + enrolling = e + if e.samples.count >= Self.planTotal { finishEnrolment() } + return + } + } + // recognition: compare each face only with enrolled people + guard !people.isEmpty else { names = []; publish(obs.map { ($0.boundingBox, nil) }); return } + var seenNames: [String] = [] + var labelled: [(CGRect, String?)] = [] + for f in obs { + // Only faces turned toward the camera are compared: feature prints are unreliable on profiles. + let frontal = abs(f.yaw?.doubleValue ?? 0) < 0.8 && abs(f.roll?.doubleValue ?? 0) < 0.6 // profiles include side samples + guard frontal, let fp = Self.print(handler, f) else { labelled.append((f.boundingBox, nil)); continue } + let scored = people.map { p -> (String, Float, Float) in + let d = p.prints.compactMap { pp -> Float? in var x: Float = 0; return (try? fp.computeDistance(&x, to: pp)) != nil ? x : nil }.min() ?? .infinity + return (p.name, d, min(p.threshold, Self.maxThreshold)) + }.sorted { $0.1 < $1.1 } + var name: String? = nil + if let b = scored.first, b.1 < b.2 { + // with several people enrolled, the best match must be clearly closer than the runner-up + if scored.count < 2 || scored[1].1 > b.1 * 1.2 { name = b.0 } + } + if let n = name, !seenNames.contains(n) { seenNames.append(n) } + labelled.append((f.boundingBox, name)) + } + names = seenNames + publish(labelled) + } + + /// Debounced state: "away" only after 60 s without a face; a people-count change must hold for 10 s. + private func settle() { + let present = Date().timeIntervalSince(lastFaceSeen) < 60 + let people = present ? max(faces, 1) : 0 + let known = present ? names.sorted() : [] + let now = Date() + if let r = reported, r.present == present, r.people == people, r.names == known { pending = nil; return } + if let p = pending, p.present == present, p.people == people, p.names == known { + guard now.timeIntervalSince(p.since) >= 10 || reported == nil else { return } + let others = max(0, people - known.count) + let who = known.isEmpty ? "" : known.joined(separator: ", ") + (others > 0 ? ", \(others) other\(others == 1 ? "" : "s")" : "") + let line = !present ? "Away from the desk" + : (people > 1 ? (who.isEmpty ? "\(people) people in view" : "In view: \(who)") + : ((facing ? "At the desk" : "At the desk, turned away") + (known.isEmpty ? "" : " · \(known[0])"))) + record?(["t": ISO8601DateFormatter().string(from: now), "app": "Camera", "bundle": "switchboard.camera", + "window": "Presence", "url": NSNull(), "via": "camera", "lines": [line]]) + reported = (present, people, known); pending = nil + } else { + pending = (present, people, known, now) + } + } + + private func publish(_ faces: [(CGRect, String?)]) { + guard viewers > 0, let cb = onFaces else { return } + let f = faces.map { (box: $0.0, name: $0.1) } + DispatchQueue.main.async { cb(f) } + } + + /// The Camera panel opened / closed: keep the camera running while anyone is looking. + func viewerOpened(_ ready: @escaping (AVCaptureSession?) -> Void) { + q.async { + self.viewers += 1 + if self.session == nil { self.start() } + let s = self.session + DispatchQueue.main.async { ready(s) } + } + } + func viewerClosed() { + q.async { + self.viewers = max(0, self.viewers - 1) + if self.viewers == 0 && !self.enabled() && self.enrolling == nil { self.stop() } + } + } + + /// Tag a specific face in the Camera panel (normalized Vision box). + func enrol(name: String, face: CGRect, progress: @escaping (Int) -> Void, prompt: @escaping (String) -> Void = { _ in }, done: @escaping (Result) -> Void) { + q.async { self.enrolTarget = face; self.enrolInto = nil } + enrol(name: name, progress: progress, prompt: prompt, done: done) + } + + /// Fix a wrong tag: this face is really `personId` (an enrolled person) — add fresh samples of it to their profile. + func correct(face: CGRect, personId: String, name: String, progress: @escaping (Int) -> Void, prompt: @escaping (String) -> Void = { _ in }, done: @escaping (Result) -> Void) { + q.async { self.enrolTarget = face; self.enrolInto = personId } + enrol(name: name, progress: progress, prompt: prompt, done: done) + } + + // MARK: enrolment + + private static func print(_ handler: VNImageRequestHandler, _ face: VNFaceObservation) -> VNFeaturePrintObservation? { + let b = face.boundingBox.insetBy(dx: -face.boundingBox.width * 0.15, dy: -face.boundingBox.height * 0.15) + .intersection(CGRect(x: 0, y: 0, width: 1, height: 1)) + let fp = VNGenerateImageFeaturePrintRequest() + fp.regionOfInterest = b + try? handler.perform([fp]) + return fp.results?.first as? VNFeaturePrintObservation + } + + /// Start enrolling someone who is present and has agreed. Uses the camera for a few seconds even if presence is off. + func enrol(name: String, progress: @escaping (Int) -> Void, prompt: @escaping (String) -> Void = { _ in }, done: @escaping (Result) -> Void) { + q.async { + let temporary = self.session == nil + if temporary { self.start() } + guard self.session != nil else { + DispatchQueue.main.async { done(.failure(NSError(domain: "camera", code: 1, userInfo: [NSLocalizedDescriptionKey: "The camera isn't available. Allow Switchboard in System Settings → Privacy → Camera."]))) } + return + } + self.enrolling = Enrol(name: name, temporary: temporary, done: { r in DispatchQueue.main.async { done(r) } }, + progress: progress, prompt: prompt) + DispatchQueue.main.async { prompt(Self.plan[0].say) } + self.q.asyncAfter(deadline: .now() + 45) { // didn't get every angle in time: give up cleanly + if let e = self.enrolling, e.samples.count < Self.planTotal { + self.enrolling = nil; self.enrolTarget = nil + if e.temporary && !self.enabled() && self.viewers == 0 { self.stop() } + e.done(.failure(NSError(domain: "camera", code: 2, userInfo: [NSLocalizedDescriptionKey: "Didn't get every angle in time (\(e.samples.count)/\(Self.planTotal)). Try again and follow the prompts slowly."]))) + } + } + } + } + + private func finishEnrolment() { + guard let e = enrolling else { return } + enrolling = nil; enrolTarget = nil + let into = enrolInto; enrolInto = nil + // threshold from how much this person's own samples vary: mean + 3·sd of pairwise distances, bounded + // consistency + threshold come from the straight-on samples only (different angles are SUPPOSED to differ) + let straight = zip(e.samples, e.poses).filter { $0.1 == "straight" }.map(\.0) + let group = straight.count >= 2 ? straight : e.samples + var ds: [Float] = [] + for i in 0.. 0.35 { + if e.temporary && !enabled() && viewers == 0 { stop() } + godLog("camera presence: enrolment of \(e.name) rejected (samples too different, mean \(String(format: "%.2f", mean)))") + e.done(.failure(NSError(domain: "camera", code: 3, userInfo: [NSLocalizedDescriptionKey: + "\(e.name)'s samples were too different from each other. Ask them to hold still and face the camera, then tag again."]))) + return + } + let sd = ds.isEmpty ? 0.1 : sqrt(ds.map { ($0 - mean) * ($0 - mean) }.reduce(0, +) / Float(ds.count)) + let threshold = min(min(max(mean + 3 * sd, mean * 1.3), mean * 2.2), Self.maxThreshold) + var id = e.name.lowercased().replacingOccurrences(of: #"[^a-z0-9]+"#, with: "-", options: .regularExpression) + "-" + String(UUID().uuidString.prefix(6)).lowercased() + var prints = e.samples.compactMap { try? NSKeyedArchiver.archivedData(withRootObject: $0, requiringSecureCoding: true).base64EncodedString() } + var consentedAt = ISO8601DateFormatter().string(from: Date()) + var finalThreshold = threshold + // correcting a tag: merge into the existing person (keep their consent time; widen only as far as needed) + if let into, let d = FileManager.default.contents(atPath: (Self.facesDir as NSString).appendingPathComponent("\(into).json")), + let old = try? JSONSerialization.jsonObject(with: d) as? [String: Any] { + id = into + prints = ((old["prints"] as? [String]) ?? []) + prints + consentedAt = old["consentedAt"] as? String ?? consentedAt + finalThreshold = min(max(Float(old["threshold"] as? Double ?? 0), threshold), Self.maxThreshold) + } + let rec: [String: Any] = ["id": id, "name": e.name, "consentedAt": consentedAt, + "prints": prints, "threshold": finalThreshold] + try? FileManager.default.createDirectory(atPath: Self.facesDir, withIntermediateDirectories: true, attributes: [.posixPermissions: 0o700]) + if let d = try? JSONSerialization.data(withJSONObject: rec) { + FileManager.default.createFile(atPath: (Self.facesDir as NSString).appendingPathComponent("\(id).json"), contents: d, attributes: [.posixPermissions: 0o600]) + } + loadPeople() + if e.temporary && !enabled() && viewers == 0 { stop() } + godLog("camera presence: enrolled \(e.name) (threshold \(String(format: "%.3f", threshold)))") + e.done(.success(e.name)) + } + + private func loadPeople() { + peopleLoaded = true + let files = (try? FileManager.default.contentsOfDirectory(atPath: Self.facesDir)) ?? [] + people = files.filter { $0.hasSuffix(".json") }.compactMap { f in + guard let d = FileManager.default.contents(atPath: (Self.facesDir as NSString).appendingPathComponent(f)), + let j = try? JSONSerialization.jsonObject(with: d) as? [String: Any], + let id = j["id"] as? String, let name = j["name"] as? String, let enc = j["prints"] as? [String] else { return nil } + let prints = enc.compactMap { Data(base64Encoded: $0) }.compactMap { + try? NSKeyedUnarchiver.unarchivedObject(ofClass: VNFeaturePrintObservation.self, from: $0) } + return Person(id: id, name: name, consentedAt: j["consentedAt"] as? String ?? "", prints: prints, + threshold: Float(j["threshold"] as? Double ?? 0.6)) + } + } + + /// For the Journal's People row (main thread). + func listPeople(_ cb: @escaping ([(id: String, name: String, consentedAt: String)]) -> Void) { + q.async { self.loadPeople(); let l = self.people.map { ($0.id, $0.name, $0.consentedAt) }; DispatchQueue.main.async { cb(l) } } + } + + func forget(personId: String, done: @escaping () -> Void) { + q.async { + try? FileManager.default.removeItem(atPath: (Self.facesDir as NSString).appendingPathComponent("\(personId).json")) + self.loadPeople() + DispatchQueue.main.async(execute: done) + } + } +} diff --git a/packages/menubar/Info.plist b/packages/menubar/Info.plist index 72890c0..c3fde9e 100644 --- a/packages/menubar/Info.plist +++ b/packages/menubar/Info.plist @@ -13,7 +13,7 @@ CFBundlePackageType APPL CFBundleShortVersionString - 0.4.0 + 0.4.1 CFBundleURLTypes @@ -26,13 +26,15 @@ CFBundleVersion - 42 + 43 LSMinimumSystemVersion 13.0 LSUIElement NSAppleEventsUsageDescription God opens apps, types, and presses the keys you approve — every action asks first, in the notch. If you opt in, Switchboard also reads your active browser tab's address locally to suggest relevant tools. Nothing leaves your Mac unasked. + NSCameraUsageDescription + Switchboard uses the camera only if you turn on camera presence in your journal settings. It notices whether you are at your desk and how many people are in view, on this Mac. No images are saved. NSHumanReadableCopyright Switchboard — BYO-Claude NSMicrophoneUsageDescription diff --git a/packages/menubar/LooseNotch.swift b/packages/menubar/LooseNotch.swift index 56b36e4..a02d393 100644 --- a/packages/menubar/LooseNotch.swift +++ b/packages/menubar/LooseNotch.swift @@ -67,6 +67,10 @@ final class LooseNotch { func open() { onOpen() } + // Where the eyes are on screen, so they can look at the pointer. + nonisolated func notchCenter() -> CGPoint? { MainActor.assumeIsolated { let f = notchFrame(); return f == .zero ? nil : CGPoint(x: f.midX, y: f.midY) } } + nonisolated func creatureCenter() -> CGPoint? { MainActor.assumeIsolated { panel.map { CGPoint(x: $0.frame.midX, y: $0.frame.midY) } } } + func dock() { panel?.orderOut(nil) model?.creatureLoose = false @@ -146,18 +150,16 @@ struct CreatureView: View { let shape = Capsule() ZStack { shape.fill(Color.page) - NotchField(accent: tint, working: model.working, animated: model.running, eyes: true) + NotchField(accent: tint, working: model.working, animated: model.running, eyes: true, + eyeColor: model.cameraOn ? Color.senseCamera : (model.listening ? Color.senseAudio : nil), + gaze: { loose.creatureCenter() }) .padding(.horizontal, 6).padding(.vertical, 3) .clipShape(shape) shape.stroke(tint.opacity(hovering ? 0.55 : 0.22), lineWidth: hovering ? 1.1 : 0.75) if hovering && !dragging { // The pill is small: icon buttons, with tooltips, instead of the notch's word pills. HStack(spacing: 8) { - if model.journalOn { - CreatureIconButton(symbol: "eye.slash", help: "Stop watching (pause the screen journal)") { - try? FileManager.default.removeItem(atPath: (NSHomeDirectory() as NSString).appendingPathComponent(".relay/journal-on")) - } - } + SensesPill() CreatureIconButton(symbol: "arrow.up.to.line", help: "Put it back in the notch") { loose.dock() } } } diff --git a/packages/menubar/OSShellView.swift b/packages/menubar/OSShellView.swift index 24b23e5..85c9556 100644 --- a/packages/menubar/OSShellView.swift +++ b/packages/menubar/OSShellView.swift @@ -150,7 +150,7 @@ struct IsoTile: View { enum Surface: String, CaseIterable, Identifiable { case home, tasks, calendar, bank // WORKSPACE case dashboard, needs, routines, workflows // AUTOMATE - case history, graph, dictionary // KNOWLEDGE + case history, journal, graph, dictionary // KNOWLEDGE case apps, store // DO var id: String { rawValue } @@ -160,7 +160,7 @@ enum Surface: String, CaseIterable, Identifiable { case .calendar: return "Calendar"; case .bank: return "Bank" case .dashboard: return "Dashboard"; case .needs: return "Needs attention" case .routines: return "Routines"; case .workflows: return "Workflows" - case .history: return "History"; case .graph: return "Graph" + case .history: return "History"; case .journal: return "Journal"; case .graph: return "Graph" case .dictionary: return "Dictionary"; case .apps: return "Apps" case .store: return "Store" } @@ -170,7 +170,7 @@ enum Surface: String, CaseIterable, Identifiable { switch self { case .home: return "⌂"; case .tasks: return "✓"; case .calendar: return "▦" case .bank: return "▤"; case .dashboard: return "▚"; case .needs: return "!" - case .routines: return "⟳"; case .workflows: return "⇉"; case .history: return "⟲" + case .routines: return "⟳"; case .workflows: return "⇉"; case .history: return "⟲"; case .journal: return "◉" case .graph: return "⊹"; case .dictionary: return "Aa"; case .apps: return "▥" case .store: return "+" } @@ -187,6 +187,7 @@ enum Surface: String, CaseIterable, Identifiable { case .routines: return "Manage and monitor the things that run without me." case .workflows: return "Run and manage my multi-step pipelines — the reusable batch recipes." case .history: return "Let me find and reopen anything I did — every God/wrapp run as a receipt." + case .journal: return "Show me what I saw and heard, as a timeline I can search and forget." case .graph: return "Show how everything connects, so I can navigate by relationship, not just by list." case .dictionary: return "Hold what my words mean — so every surface and wrapp speaks my language." case .apps: return "Launch, manage, and understand the tools I have." @@ -205,6 +206,7 @@ enum Surface: String, CaseIterable, Identifiable { case .routines: return ["Active", "Paused", "Last run", "Next run", "Logs"] case .workflows: return ["Pipelines", "Steps", "Runs", "Templates"] case .history: return ["Today", "Yesterday", "Filters", "Reopen"] + case .journal: return ["Day strip", "Sessions", "Search", "Forget"] case .graph: return ["Canvas", "Filters", "Inspector", "List view"] case .dictionary: return ["A–Z index", "Term", "Definition", "Scope", "Source"] case .apps: return ["Pinned", "All apps", "Categories", "God can drive"] @@ -217,7 +219,7 @@ struct RailGroup: Identifiable { let name: String; let items: [Surface]; var id: let OS_GROUPS: [RailGroup] = [ RailGroup(name: "Workspace", items: [.home, .tasks, .calendar, .bank]), RailGroup(name: "Automate", items: [.dashboard, .needs, .routines, .workflows]), - RailGroup(name: "Knowledge", items: [.history, .graph, .dictionary]), + RailGroup(name: "Knowledge", items: [.history, .journal, .graph, .dictionary]), RailGroup(name: "Do", items: [.apps, .store]), ] @@ -607,6 +609,7 @@ struct OSShellView: View { case .routines: RoutinesSurface(onNavigate: { selected = $0 }) case .workflows: WorkflowsSurface(onNavigate: { selected = $0 }) case .history: HistorySurface(onNavigate: { selected = $0 }) + case .journal: JournalSurface(onNavigate: { selected = $0 }) case .graph: GraphSurface(onNavigate: { selected = $0 }) case .dictionary: DictionarySurface(onNavigate: { selected = $0 }) case .apps: AppsSurface(onNavigate: { selected = $0 }) diff --git a/packages/menubar/OSSurfaceJournal.swift b/packages/menubar/OSSurfaceJournal.swift new file mode 100644 index 0000000..5702d4d --- /dev/null +++ b/packages/menubar/OSSurfaceJournal.swift @@ -0,0 +1,530 @@ +// OSSurfaceJournal.swift — the JOURNAL surface of Switchboard OS (Knowledge group, next to History). +// +// Reads the local screen journal (~/.relay/journal/YYYY-MM-DD.jsonl, written by ScreenJournal.swift and +// CallAudio.swift) and shows one day as: +// • a DAY STRIP: the hours of the day with a coloured segment per app session (calls outlined amber); click jumps +// • a SESSION TIMELINE: consecutive snapshots of the same window merged into sessions; each opens to its text, +// and a call opens as a conversation (Me / Them bubbles) +// • SEARCH across the day (local, instant; nothing leaves the Mac), app filter chips +// • CONTROLS: recording on/off, forget a session, forget the last hour +// Same design law as the other surfaces: graphite chrome, lime = actionable, colour only in the app marks. + +import SwiftUI + +struct JEntry { + let t: Date + let app: String + let bundle: String + let window: String + let via: String + let call: Bool + let audio: Bool + let lines: [String] +} + +struct JSession: Identifiable { + let id: String + let app: String + let bundle: String + let window: String + let start: Date + var end: Date + var call: Bool + var audio: Bool + var lines: [String] + var talk: [(who: String, text: String, t: Date)] +} + +private let journalDir = (NSHomeDirectory() as NSString).appendingPathComponent(".relay/journal") +private let journalFlag = (NSHomeDirectory() as NSString).appendingPathComponent(".relay/journal-on") +private let dayFmt: DateFormatter = { let f = DateFormatter(); f.dateFormat = "yyyy-MM-dd"; f.timeZone = TimeZone(identifier: "UTC"); return f }() +private let clockFmt: DateFormatter = { let f = DateFormatter(); f.dateFormat = "HH:mm"; return f }() +private let iso = ISO8601DateFormatter() + +/// Window titles drift within one meeting ("Meet – abc-defg-hij – Microphone…" vs "Meet – abc-defg-hij 🔊"): +/// group by the Meet code when there is one. +private func sessionKey(_ e: JEntry) -> String { + let w = e.window.lowercased() + if let r = w.range(of: #"[a-z]{3}-[a-z]{4}-[a-z]{3}"#, options: .regularExpression) { return e.bundle + "|meet:" + w[r] } + return e.bundle + "|" + e.window +} + +func journalDays() -> [String] { + ((try? FileManager.default.contentsOfDirectory(atPath: journalDir)) ?? []) + .filter { $0.hasSuffix(".jsonl") }.map { String($0.prefix(10)) }.sorted(by: >) +} + +func loadJournal(day: String) -> [JEntry] { + let path = (journalDir as NSString).appendingPathComponent("\(day).jsonl") + guard let text = try? String(contentsOfFile: path, encoding: .utf8) else { return [] } + return text.split(separator: "\n").compactMap { row in + guard let j = try? JSONSerialization.jsonObject(with: Data(row.utf8)) as? [String: Any], + let ts = j["t"] as? String, let t = iso.date(from: ts) else { return nil } + return JEntry(t: t, app: j["app"] as? String ?? "", bundle: j["bundle"] as? String ?? "", + window: j["window"] as? String ?? "", via: j["via"] as? String ?? "", + call: j["call"] as? Bool ?? false, audio: (j["source"] as? String) == "audio", + lines: j["lines"] as? [String] ?? []) + }.sorted { $0.t < $1.t } +} + +private func isCallKey(_ key: String) -> Bool { key.contains("|meet:") } +private func isCallEntry(_ e: JEntry) -> Bool { e.call || e.audio || isCallKey(sessionKey(e)) } + +/// Sessions for one day. A CALL is one session per meeting (grouped by its Meet code / meeting window across the +/// whole day, split only by a gap over 30 min), however often you switched away during it. Everything else merges +/// consecutive snapshots of the same window (≤ 10 min apart). +func journalSessions(_ entries: [JEntry]) -> [JSession] { + var out: [JSession] = [] + func add(_ e: JEntry, into s: inout JSession) { + s.end = max(s.end, e.t); s.call = s.call || isCallEntry(e); s.audio = s.audio || e.audio + if e.audio { s.talk += e.lines.map { talkLine($0, e.t) } } else { s.lines += e.lines } + } + func new(_ e: JEntry, _ key: String) -> JSession { + var s = JSession(id: "\(key)@\(iso.string(from: e.t))", app: e.app, bundle: e.bundle, window: e.window, + start: e.t, end: e.t, call: isCallEntry(e), audio: false, lines: [], talk: []) + add(e, into: &s); return s + } + // calls: by meeting, across the day + var openCall: [String: Int] = [:] + for e in entries where isCallEntry(e) { + let key = sessionKey(e) + if let i = openCall[key], e.t.timeIntervalSince(out[i].end) < 1800 { add(e, into: &out[i]) } + else { out.append(new(e, key)); openCall[key] = out.count - 1 } + } + // everything else: consecutive snapshots of the same window + var lastKey = "", lastIdx = -1 + for e in entries where !isCallEntry(e) { + let key = sessionKey(e) + if key == lastKey, lastIdx >= 0, e.t.timeIntervalSince(out[lastIdx].end) < 600 { add(e, into: &out[lastIdx]) } + else { out.append(new(e, key)); lastIdx = out.count - 1 } + lastKey = key + } + return out.sorted { $0.start < $1.start } +} + +/// "Me: …", "Them: …", or a caption speaker "Bhoomi Jain: …". +private func talkLine(_ l: String, _ t: Date) -> (who: String, text: String, t: Date) { + if let r = l.range(of: ": "), l.distance(from: l.startIndex, to: r.lowerBound) <= 32 { + let who = String(l[l.startIndex.. Bool) { + let path = (journalDir as NSString).appendingPathComponent("\(day).jsonl") + guard let text = try? String(contentsOfFile: path, encoding: .utf8) else { return } + let keep = text.split(separator: "\n").filter { row in + guard let j = try? JSONSerialization.jsonObject(with: Data(row.utf8)) as? [String: Any], + let ts = j["t"] as? String, let t = iso.date(from: ts) else { return true } + let e = JEntry(t: t, app: j["app"] as? String ?? "", bundle: j["bundle"] as? String ?? "", + window: j["window"] as? String ?? "", via: "", call: false, audio: false, lines: []) + return !drop(e, j) + } + let body = keep.isEmpty ? "" : keep.joined(separator: "\n") + "\n" + FileManager.default.createFile(atPath: path, contents: Data(body.utf8), attributes: [.posixPermissions: 0o600]) +} + +// ===================================================================================================== + +struct JournalSurface: View { + var onNavigate: (Surface) -> Void = { _ in } + + @State private var days: [String] = [] + @State private var day = dayFmt.string(from: Date()) + @State private var sessions: [JSession] = [] + @State private var open: Set = [] + @State private var q = "" + @State private var appFilter: String? = nil + @State private var recording = FileManager.default.fileExists(atPath: journalFlag) + @State private var confirmForget: String? = nil // session id, or "hour" + @State private var loadedStamp: Date = .distantPast + @State private var lastLoad: Date = .distantPast + + private var apps: [String] { + var seen: [String] = []; for s in sessions where !seen.contains(s.app) { seen.append(s.app) }; return seen + } + private var visible: [JSession] { + let ql = q.trimmingCharacters(in: .whitespaces).lowercased() + return sessions.filter { s in + (appFilter == nil || s.app == appFilter) && + (ql.isEmpty || (s.app + " " + s.window + " " + s.lines.joined(separator: " ") + " " + s.talk.map(\.text).joined(separator: " ")).lowercased().contains(ql)) + } + } + + var body: some View { + ScrollViewReader { proxy in + ScrollView { + VStack(alignment: .leading, spacing: 0) { + header + if Senses.cameraLab { PeoplePanel().padding(.top, 14) } + if sessions.isEmpty { + emptyState + } else { + DayStrip(sessions: sessions, highlight: Set(visible.map(\.id))) { id in + open.insert(id) + withAnimation { proxy.scrollTo(id, anchor: .top) } + } + .padding(.top, 18) + chips.padding(.top, 14) + if visible.isEmpty { + Text("Nothing on \(dayLabel) matches \"\(q)\".") + .font(.hanken(12.5)).foregroundColor(.inkDim) + .frame(maxWidth: .infinity).padding(24) + .background(RoundedRectangle(cornerRadius: 12).strokeBorder(Color.edge, style: StrokeStyle(lineWidth: 1, dash: [4, 4]))) + .padding(.top, 18) + } + VStack(alignment: .leading, spacing: 8) { + ForEach(visible) { s in + JournalSessionRow(session: s, open: open.contains(s.id), query: q, + confirming: confirmForget == s.id, + onToggle: { if open.contains(s.id) { open.remove(s.id) } else { open.insert(s.id) } }, + onForget: { forgetSession(s) }) + .id(s.id) + } + } + .padding(.top, 16) + Text("kept on this Mac in ~/.relay/journal · days older than 30 are deleted · search here never leaves the Mac") + .font(.splMono(10.5)).foregroundColor(.inkFaint).padding(.top, 28) + } + } + .padding(.horizontal, 28).padding(.top, 8).padding(.bottom, 48) + .frame(maxWidth: .infinity, alignment: .leading) + } + } + .onAppear { reload(force: true) } + .onReceive(OSPulse.shared.$tick) { _ in reload(force: false) } + } + + private var dayLabel: String { + day == dayFmt.string(from: Date()) ? "today" : day + } + + private var header: some View { + VStack(alignment: .leading, spacing: 12) { + HStack(alignment: .firstTextBaseline, spacing: 12) { + VStack(alignment: .leading, spacing: 6) { + Text("JOURNAL").font(.splMono(10.5)).tracking(1.8).foregroundColor(.inkFaint) + Text(day == dayFmt.string(from: Date()) ? "What you saw today" : "What you saw on \(day)") + .font(.hanken(24, .semibold)).foregroundColor(.ink) + } + Spacer(minLength: 0) + SensesPill(compact: false) + JChip(text: confirmForget == "hour" ? "Confirm: forget last hour" : "Forget last hour", + active: confirmForget == "hour", tint: confirmForget == "hour" ? .danger : .inkSec) { forgetHour() } + } + HStack(spacing: 8) { + TextField("search \(dayLabel)… (names, topics, what someone said)", text: $q) + .textFieldStyle(.plain) + .font(.splMono(12)).foregroundColor(.ink) + .padding(.horizontal, 12).padding(.vertical, 7) + .background(RoundedRectangle(cornerRadius: 8).fill(Color(red: 0.06, green: 0.07, blue: 0.09))) + .overlay(RoundedRectangle(cornerRadius: 8).stroke(Color.edge, lineWidth: 1)) + .frame(maxWidth: 380, alignment: .leading) + Spacer(minLength: 0) + ForEach(days.prefix(7), id: \.self) { d in + JChip(text: d == dayFmt.string(from: Date()) ? "Today" : String(d.dropFirst(5)), active: d == day) { + day = d; open = []; reload(force: true) + } + } + } + } + } + + private var chips: some View { + ScrollView(.horizontal, showsIndicators: false) { + HStack(spacing: 6) { + JChip(text: "All apps", active: appFilter == nil) { appFilter = nil } + ForEach(apps, id: \.self) { a in + JChip(text: a, active: appFilter == a, tint: colorForId(a)) { appFilter = appFilter == a ? nil : a } + } + } + } + } + + private var emptyState: some View { + VStack(alignment: .leading, spacing: 10) { + Text(recording ? "Nothing journaled on \(dayLabel) yet." : "The journal is paused.") + .font(.brico(20, .bold)).foregroundColor(.ink) + Text("While it records, Switchboard keeps a private text journal of what's on your screen (and, if you allow it, what's said in meetings). It stays on this Mac. Password managers, private messages, banking and key pages are never read.") + .font(.hanken(13)).foregroundColor(.inkSec).fixedSize(horizontal: false, vertical: true) + if !recording { + LimeButton(label: "Start recording") { FileManager.default.createFile(atPath: journalFlag, contents: nil); recording = true } + } + } + .padding(22).frame(maxWidth: .infinity, alignment: .leading) + .background(RoundedRectangle(cornerRadius: 16).fill(Color.panel)) + .overlay(RoundedRectangle(cornerRadius: 16).stroke(Color.edge, lineWidth: 1)) + .padding(.top, 20) + } + + // The OS pulse fires on any ~/.relay change and the journal writes every few seconds: reload only when the + // day's file changed, and at most every 5 s. + private func reload(force: Bool) { + recording = FileManager.default.fileExists(atPath: journalFlag) + let path = (journalDir as NSString).appendingPathComponent("\(day).jsonl") + let stamp = (try? FileManager.default.attributesOfItem(atPath: path)[.modificationDate] as? Date) ?? .distantPast + guard force || (stamp != loadedStamp && Date().timeIntervalSince(lastLoad) > 5) else { return } + loadedStamp = stamp; lastLoad = Date() + days = journalDays() + let d = day + DispatchQueue.global(qos: .userInitiated).async { + let s = journalSessions(loadJournal(day: d)).reversed() // newest first + DispatchQueue.main.async { if d == day { sessions = Array(s) } } + } + } + + private func forgetSession(_ s: JSession) { + guard confirmForget == s.id else { confirmForget = s.id; return } + confirmForget = nil + let key = sessionKey(JEntry(t: s.start, app: s.app, bundle: s.bundle, window: s.window, via: "", call: false, audio: false, lines: [])) + forgetJournal(day: day) { e, _ in + e.t >= s.start && e.t <= s.end && (sessionKey(e) == key || e.bundle == s.bundle) + } + reload(force: true) + } + + private func forgetHour() { + guard confirmForget == "hour" else { confirmForget = "hour"; return } + confirmForget = nil + let cutoff = Date().addingTimeInterval(-3600) + forgetJournal(day: dayFmt.string(from: Date())) { e, _ in e.t >= cutoff } + forgetJournal(day: dayFmt.string(from: cutoff)) { e, _ in e.t >= cutoff } + reload(force: true) + } +} + +// ---- the day strip: hours across, one lane, a segment per session ---- +private struct DayStrip: View { + let sessions: [JSession] + let highlight: Set + let onPick: (String) -> Void + @State private var hover: String? = nil + + var body: some View { + let start = Calendar.current.dateInterval(of: .hour, for: sessions.map(\.start).min() ?? Date())?.start ?? Date() + let endRaw = sessions.map(\.end).max() ?? Date() + let end = Calendar.current.dateInterval(of: .hour, for: endRaw)?.end ?? endRaw + let span = max(end.timeIntervalSince(start), 3600) + VStack(alignment: .leading, spacing: 6) { + GeometryReader { geo in + let w = geo.size.width + ZStack(alignment: .topLeading) { + RoundedRectangle(cornerRadius: 6).fill(Color.panel) + .overlay(RoundedRectangle(cornerRadius: 6).stroke(Color.edge, lineWidth: 1)) + ForEach(sessions) { s in + let x = CGFloat(s.start.timeIntervalSince(start) / span) * w + let sw = max(3, CGFloat(max(s.end.timeIntervalSince(s.start), 30) / span) * w) + // top lane = apps, bottom lane = calls (a call spans its whole meeting) + RoundedRectangle(cornerRadius: 2) + .fill((s.call ? Color.senseAudio : colorForId(s.app)).opacity(highlight.contains(s.id) ? 0.9 : 0.25)) + .frame(width: sw, height: hover == s.id ? 20 : 16) + .offset(x: x, y: s.call ? 30 : (hover == s.id ? 4 : 6)) + .onHover { hover = $0 ? s.id : (hover == s.id ? nil : hover) } + .onTapGesture { onPick(s.id) } + .help("\(clockFmt.string(from: s.start))–\(clockFmt.string(from: s.end)) · \(s.app) · \(s.window)") + } + } + } + .frame(height: 52) + HStack { + Text(clockFmt.string(from: start)).font(.splMono(10)).foregroundColor(.inkFaint) + Spacer() + if let h = hover, let s = sessions.first(where: { $0.id == h }) { + Text("\(clockFmt.string(from: s.start))–\(clockFmt.string(from: s.end)) · \(s.app) · \(String(s.window.prefix(50)))") + .font(.splMono(10)).foregroundColor(.inkSec) + Spacer() + } + Text(clockFmt.string(from: end)).font(.splMono(10)).foregroundColor(.inkFaint) + } + } + } +} + +// ---- one session: header row, opens to its text or its conversation ---- +private struct JournalSessionRow: View { + let session: JSession + let open: Bool + let query: String + let confirming: Bool + let onToggle: () -> Void + let onForget: () -> Void + @State private var hover = false + @State private var showAll = false + + var body: some View { + let s = session + VStack(alignment: .leading, spacing: 10) { + Button(action: onToggle) { + HStack(alignment: .center, spacing: 12) { + RoundedRectangle(cornerRadius: 3).fill(colorForId(s.app)).frame(width: 4, height: 30) + VStack(alignment: .leading, spacing: 3) { + HStack(spacing: 8) { + Text(s.app).font(.hanken(13.5, .semibold)).foregroundColor(.ink) + if s.call { Tag(text: s.audio ? "call · transcript" : "call", color: .senseAudio) } + } + Text(s.window.isEmpty ? "—" : s.window).font(.hanken(12)).foregroundColor(.inkSec).lineLimit(1) + } + Spacer(minLength: 8) + Text("\(s.lines.count + s.talk.count) lines").font(.splMono(10.5)).foregroundColor(.inkFaint) + Text(s.start == s.end ? clockFmt.string(from: s.start) : "\(clockFmt.string(from: s.start))–\(clockFmt.string(from: s.end))") + .font(.splMono(11)).foregroundColor(.inkSec).frame(width: 92, alignment: .trailing) + Text("▾").font(.splMono(9)).foregroundColor(.inkFaint).rotationEffect(.degrees(open ? 0 : -90)) + } + .contentShape(Rectangle()) + } + .buttonStyle(.plain) + + if open { + if !s.talk.isEmpty { conversation(s) } + if !s.lines.isEmpty { screenText(s) } + HStack { + Spacer() + JChip(text: confirming ? "Confirm: forget this session" : "Forget this session", + active: confirming, tint: confirming ? .danger : .inkDim, action: onForget) + } + } + } + .padding(.horizontal, 14).padding(.vertical, 11) + .background(RoundedRectangle(cornerRadius: 12).fill(hover || open ? Color.panel : Color.clear)) + .overlay(RoundedRectangle(cornerRadius: 12).stroke(open ? Color.edge : Color.edgeSoft, lineWidth: 1)) + .onHover { hover = $0 } + } + + private func conversation(_ s: JSession) -> some View { + VStack(alignment: .leading, spacing: 6) { + Text("WHAT WAS SAID").font(.splMono(9.5)).tracking(1.4).foregroundColor(.inkFaint) + ForEach(Array(s.talk.enumerated()), id: \.offset) { _, t in + HStack(alignment: .top) { + if t.who == "Me" { Spacer(minLength: 60) } + VStack(alignment: t.who == "Me" ? .trailing : .leading, spacing: 3) { + Text("\(t.who.isEmpty ? "?" : t.who) · \(clockFmt.string(from: t.t))").font(.splMono(9.5)).foregroundColor(.inkFaint) + Text(t.text).font(.hanken(12.5)).foregroundColor(.ink) + .fixedSize(horizontal: false, vertical: true).textSelection(.enabled) + } + .padding(.horizontal, 10).padding(.vertical, 7) + .background(RoundedRectangle(cornerRadius: 10).fill(t.who == "Me" ? Color.lime.opacity(0.10) : Color(red: 0.08, green: 0.09, blue: 0.11))) + .overlay(RoundedRectangle(cornerRadius: 10).stroke(t.who == "Me" ? Color.lime.opacity(0.35) : Color.edge, lineWidth: 1)) + if t.who != "Me" { Spacer(minLength: 60) } + } + } + } + } + + private func screenText(_ s: JSession) -> some View { + let ql = query.trimmingCharacters(in: .whitespaces).lowercased() + let lines = ql.isEmpty ? s.lines : s.lines.filter { $0.lowercased().contains(ql) } + let shown = showAll ? lines : Array(lines.prefix(40)) + return VStack(alignment: .leading, spacing: 4) { + Text(ql.isEmpty ? "ON SCREEN" : "ON SCREEN · MATCHES").font(.splMono(9.5)).tracking(1.4).foregroundColor(.inkFaint) + ForEach(Array(shown.enumerated()), id: \.offset) { _, l in + Text(l).font(.splMono(11.5)).foregroundColor(.inkSec).lineLimit(3).textSelection(.enabled) + } + if lines.count > shown.count { + Button("Show all \(lines.count) lines") { showAll = true } + .buttonStyle(.plain).font(.hanken(12, .semibold)).foregroundColor(.lime) + } + } + } +} + +private struct Tag: View { + let text: String + let color: Color + var body: some View { + Text(text).font(.splMono(9.5)).foregroundColor(color) + .padding(.horizontal, 6).padding(.vertical, 2) + .overlay(Capsule().stroke(color.opacity(0.6), lineWidth: 1)) + } +} + +/// Filter / action chip (same look as the Knowledge surfaces' chips). +private struct JChip: View { + let text: String + var active: Bool = false + var tint: Color = .inkSec + let action: () -> Void + @State private var hover = false + var body: some View { + Button(action: action) { + Text(text).font(.hanken(12, active ? .semibold : .regular)) + .foregroundColor(active ? (tint == .inkSec ? .ink : tint) : tint) + .padding(.horizontal, 11).padding(.vertical, 6) + .background(RoundedRectangle(cornerRadius: 8).fill(Color.panel)) + .overlay(RoundedRectangle(cornerRadius: 8).stroke((hover || active) ? tint.opacity(0.6) : Color.edge, lineWidth: 1)) + } + .buttonStyle(.plain) + .onHover { hover = $0 } + } +} + + +// ---- People the camera may recognise: enrolled only with the person present and agreeing (CameraPresence) ---- +private struct PeoplePanel: View { + @State private var people: [(id: String, name: String, consentedAt: String)] = [] + @State private var adding = false + @State private var name = "" + @State private var consent = false + @State private var progress: Int? = nil + @State private var message: String? = nil + @State private var confirmRemove: String? = nil + + var body: some View { + VStack(alignment: .leading, spacing: 8) { + HStack(spacing: 6) { + Text("PEOPLE THE CAMERA KNOWS").font(.splMono(9.5)).tracking(1.4).foregroundColor(.inkFaint) + if people.isEmpty { Text("· nobody yet; everyone else is only counted").font(.splMono(9.5)).foregroundColor(.inkFaint) } + Spacer(minLength: 0) + } + HStack(spacing: 6) { + ForEach(people, id: \.id) { p in + JChip(text: confirmRemove == p.id ? "Remove \(p.name)?" : p.name, active: confirmRemove == p.id, + tint: confirmRemove == p.id ? .danger : .senseCamera) { + if confirmRemove == p.id { CameraPresence.shared.forget(personId: p.id) { confirmRemove = nil; reload() } } + else { confirmRemove = p.id } + } + .help("Enrolled \(String(p.consentedAt.prefix(10))) with their consent. Click twice to remove their face data.") + } + if !adding { JChip(text: "+ Add a person", tint: .inkSec) { adding = true; message = nil } } + JChip(text: "Open camera", tint: .senseCamera) { CameraPanelController.shared.show() } + } + if adding { + HStack(spacing: 10) { + TextField("their name", text: $name) + .textFieldStyle(.plain).font(.hanken(12.5)).foregroundColor(.ink) + .padding(.horizontal, 10).padding(.vertical, 6) + .background(RoundedRectangle(cornerRadius: 8).fill(Color(red: 0.06, green: 0.07, blue: 0.09))) + .overlay(RoundedRectangle(cornerRadius: 8).stroke(Color.edge, lineWidth: 1)) + .frame(width: 170) + Toggle(isOn: $consent) { + Text("They're here and agree to be recognised on this Mac").font(.hanken(12)).foregroundColor(.inkSec) + } + .toggleStyle(.checkbox) + JChip(text: progress.map { "Follow the prompts… \($0)/\(CameraPresence.planTotal)" } ?? "Capture", + active: consent && !name.trimmingCharacters(in: .whitespaces).isEmpty, + tint: .senseCamera) { capture() } + .disabled(!consent || name.trimmingCharacters(in: .whitespaces).isEmpty || progress != nil) + JChip(text: "Cancel", tint: .inkDim) { adding = false; name = ""; consent = false } + } + } + if let m = message { Text(m).font(.hanken(12)).foregroundColor(.inkSec) } + } + .onAppear(perform: reload) + } + + private func reload() { CameraPresence.shared.listPeople { people = $0 } } + + private func capture() { + let n = name.trimmingCharacters(in: .whitespaces) + progress = 0 + CameraPresence.shared.enrol(name: n, progress: { progress = $0 }) { r in + progress = nil + switch r { + case .success(let who): message = "Enrolled \(who). The camera will recognise them from now on."; adding = false; name = ""; consent = false + case .failure(let e): message = e.localizedDescription + } + reload() + } + } +} diff --git a/packages/menubar/Relay.entitlements b/packages/menubar/Relay.entitlements index 9c7081d..6a58695 100644 --- a/packages/menubar/Relay.entitlements +++ b/packages/menubar/Relay.entitlements @@ -17,6 +17,9 @@ com.apple.security.device.audio-input + + com.apple.security.device.camera + com.apple.security.automation.apple-events diff --git a/packages/menubar/RelayMenuBar.swift b/packages/menubar/RelayMenuBar.swift index 7a1f774..a0909d1 100644 --- a/packages/menubar/RelayMenuBar.swift +++ b/packages/menubar/RelayMenuBar.swift @@ -195,6 +195,9 @@ extension Color { static let inkDim = Color(red: 0x9A/255.0, green: 0x9A/255.0, blue: 0xA2/255.0) // neutral, no blue static let inkFaint = Color(red: 0x6C/255.0, green: 0x6C/255.0, blue: 0x74/255.0) static let lime = Color(red: 0xC8/255.0, green: 0xF2/255.0, blue: 0x50/255.0) + // Eye colours = what Switchboard can sense: lime = screen, amber = screen + audio, coral = + camera (reserved). + static let senseAudio = Color(red: 0xFF/255.0, green: 0xB5/255.0, blue: 0x47/255.0) + static let senseCamera = Color(red: 0xFF/255.0, green: 0x6B/255.0, blue: 0x5A/255.0) static let danger = Color(red: 0xFF/255.0, green: 0x2D/255.0, blue: 0x6E/255.0) static let ok = Color(red: 0x3D/255.0, green: 0xD6/255.0, blue: 0x8C/255.0) // "connected" green static let amber = Color(red: 0xEF/255.0, green: 0x9F/255.0, blue: 0x27/255.0) // pending / needs-attention @@ -788,6 +791,8 @@ final class Model: ObservableObject { @Published var journalOn = false // screen journal recording (~/.relay/journal-on) → eyes on the notch @Published var creatureLoose = false // the Loose Notch pill is off the notch (LooseNotch.swift) → the eyes go with it @Published var regionSelect = false // ⌃⌃ lets you drag a screen region → only that is sent + @Published var listening = false // CallAudio is transcribing a call → the eyes turn amber + @Published var cameraOn = false // CameraPresence is watching → the eyes turn coral @Published var defaultShare = false // ⌃⌃ auto-shares the whole screen (fn+click then TOGGLES it off) @Published var modelProviders: [ModelProvider] = [] @Published var originModels: [String: String] = [:] // origin → its current model (grants.json modelOverride) @@ -2398,8 +2403,13 @@ struct NotchField: View { private static let eyeShape: [(Int, Int)] = [(1, 0), (2, 0), (0, 1), (1, 1), (2, 1), (3, 1), (1, 2), (2, 2)] private func eyeLamp(_ c: Int, _ r: Int, _ cols: Int, _ rows: Int, _ t: Double) -> Bool { let blink = t.truncatingRemainder(dividingBy: 4.2) < 0.16 - let glance = [0, 0, -1, 0, 1, 0][Int(t / 5.3) % 6] - let top = max(0, rows / 2 - 1) + var glance = [0, 0, -1, 0, 1, 0][Int(t / 5.3) % 6], lift = 0 + if let c = gaze?() { // look at the pointer: ±2 lamps across, ±1 up/down + let m = NSEvent.mouseLocation + glance = max(-2, min(2, Int(((m.x - c.x) / 90).rounded()))) + if rows >= 5 { lift = m.y - c.y > 40 ? -1 : (m.y - c.y < -40 ? 1 : 0) } + } + let top = max(0, min(rows - 3, rows / 2 - 1 + lift)) let left = cols / 2 - 5 + glance // two 4-wide eyes, 2 lamps apart, centred for ox in [left, left + 6] { for (dc, dr) in Self.eyeShape where !(blink && dr != 1) { @@ -2418,13 +2428,17 @@ struct NotchField: View { // Lamp density matches the dictation DotMatrix (founder-approved 2026-08-13, "chunky lamps"): // pitch 6 / dot 3 — bolder + less dense than the old fine field (pitch 4 / dot 1.8). let gap: CGFloat = 6.0, d: CGFloat = 3.0 + var eyeColor: Color? = nil // nil = the accent; amber while listening to a call + /// Screen point of this field's centre; when set, the eyes look toward the mouse pointer. + var gaze: (() -> CGPoint?)? = nil let cols = max(1, Int(size.width / gap)), rows = max(1, Int(size.height / gap)) let ox = (size.width - CGFloat(cols - 1) * gap) / 2, oy = (size.height - CGFloat(rows - 1) * gap) / 2 for c in 0.. ~/.relay/open-os`); empty = Home. let want = (try? String(contentsOfFile: p, encoding: .utf8))?.trimmingCharacters(in: .whitespacesAndNewlines) ?? "" try? FileManager.default.removeItem(atPath: p) - if !want.isEmpty, let s = Surface(rawValue: want) { OSShellWindowController.shared.show(s) } + if want == "camera" { if Senses.cameraLab { CameraPanelController.shared.show() } } // `echo camera > ~/.relay/open-os` (camera lab only) + else if !want.isEmpty, let s = Surface(rawValue: want) { OSShellWindowController.shared.show(s) } else { OSShellWindowController.shared.show() } } // `touch ~/.relay/fill-form` → guided form-fill from the clipboard (scriptable + self-test hook). diff --git a/packages/menubar/ScreenJournal.swift b/packages/menubar/ScreenJournal.swift index 1011cf5..4762896 100644 --- a/packages/menubar/ScreenJournal.swift +++ b/packages/menubar/ScreenJournal.swift @@ -9,7 +9,11 @@ // Anything that later leaves the Mac (recall, live cards) goes through a separate, redacting step. // • BLOCK LIST. Password managers, private messengers, system security UI, private/incognito windows and // banking-looking pages are never read. Secure (password) fields are never read anywhere. -// ~/.relay/journal.json {"block": [bundleIds], "blockWords": [...], "retentionDays": 30} extends it. +// ~/.relay/journal.json {"block": [bundleIds], "blockWords": [...], "retentionDays": 30, "calls": true} extends it. +// • CALL AUDIO (CallAudio.swift): with "audio": true, calls also get on-device Me / Them transcripts. +// • CALLS. Meet/Zoom/Teams/FaceTime windows are re-read every 20 s at higher OCR resolution (screen shares change +// without the title changing); "calls": false turns call capture off entirely. On every app switch the window +// being left gets one LAST LOOK, so a quick glance at a call or document is still journaled. // • DIFF, NOT DUMP. Only lines not seen recently are written; secret-looking tokens are masked on write. // • RETENTION. Days older than retentionDays (default 30) are deleted on start and daily. // @@ -30,6 +34,7 @@ final class ScreenJournal { private var seen: [String: Date] = [:] // line hash → last written (drops re-renders of the same text) private var lastPrune = Date.distantPast private var electronAsked = Set() + private var lastApp: NSRunningApplication? // the app last sampled as frontmost → its "last look" on switch private var lastOn: Bool? private var lastOCRKey = "" private var lastOCR = Date.distantPast @@ -88,12 +93,38 @@ final class ScreenJournal { godLog("screen journal OFF") } - @objc private func appActivated(_ note: Notification) { queue.asyncAfter(deadline: .now() + 1) { self.tick(force: true) } } + /// On every app switch: first a LAST LOOK at the window being left (it is still on screen, and a call or a + /// shared document can change without its title changing, so the regular re-read may never have caught it), + /// then a fresh read of the app now in front. + @objc private func appActivated(_ note: Notification) { + let next = (note.userInfo?[NSWorkspace.applicationUserInfoKey] as? NSRunningApplication)?.processIdentifier + queue.async { + if let prev = self.lastApp, prev.processIdentifier != next, !prev.isTerminated { + self.sample(prev, force: true, lastLook: true) + } + } + queue.asyncAfter(deadline: .now() + 1) { self.tick(force: true) } + } // MARK: sampling private func tick(force: Bool) { - guard running, FileManager.default.fileExists(atPath: flagPath), AXIsProcessTrusted(), let app = NSWorkspace.shared.frontmostApplication else { return } + guard let app = NSWorkspace.shared.frontmostApplication else { return } + lastApp = app + sample(app, force: force, lastLook: false) + } + + /// Video calls: screen shares and captions change without the window title changing. + private func isCall(bundle: String, title: String, url: String?) -> Bool { + let callApps: Set = ["us.zoom.xos", "com.microsoft.teams", "com.microsoft.teams2", "com.apple.facetime", "com.cisco.webexmeetingsapp"] + if callApps.contains(bundle) { return true } + let u = (url ?? "").lowercased(), t = title.lowercased() + return u.range(of: #"meet\.google\.com/[a-z]{3}-[a-z]{4}-[a-z]{3}"#, options: .regularExpression) != nil + || t.hasPrefix("meet – ") || t.hasPrefix("meet - ") || u.contains("zoom.us/wc") || u.contains("teams.microsoft.com") + } + + private func sample(_ app: NSRunningApplication, force: Bool, lastLook: Bool) { + guard running, FileManager.default.fileExists(atPath: flagPath), AXIsProcessTrusted() else { return } let bundle = (app.bundleIdentifier ?? "unknown").lowercased() let cfg = config() if cfg.block.contains(bundle) { return } @@ -107,11 +138,17 @@ final class ScreenJournal { guard let win = jElement(axApp, kAXFocusedWindowAttribute as String) else { note("skip \(bundle): no focused window"); return } let title = jString(win, kAXTitleAttribute as String) ?? "" let key = bundle + "|" + title - let stale = Date().timeIntervalSince(lastRead) > 30 - guard force || key != lastKey || stale else { return } - lastKey = key; lastRead = Date() - let url = jURL(win) + let call = isCall(bundle: bundle, title: title, url: url) + if call && !cfg.calls { CallAudio.shared.stopIfActive(reason: "calls off"); return } // "calls": false → never journal calls + // Call audio has its own background meeting watch + consent card (CallAudio.arm); nothing to do here. + let every: TimeInterval = call ? 20 : 30 + if !lastLook { + let stale = Date().timeIntervalSince(lastRead) > every + guard force || key != lastKey || stale else { return } + lastKey = key; lastRead = Date() + } + let haystack = (title + " " + (url ?? "")).lowercased() if let w = cfg.blockWords.first(where: { haystack.contains($0) }) { note("skip \(bundle): block word '\(w)'"); return } @@ -120,10 +157,11 @@ final class ScreenJournal { var via = "ax" // Thin AX text (Chrome and other apps that hide page text from accessibility) → read the pixels: // on-device Vision OCR of just this window. On a window change, and at most once a minute otherwise. - if lines.count < 8 || lines.reduce(0, { $0 + $1.count }) < 300, - key != lastOCRKey || Date().timeIntervalSince(lastOCR) > 60 { + let sinceOCR = Date().timeIntervalSince(lastOCR) + let ocrDue = lastLook ? !(key == lastOCRKey && sinceOCR < 10) : (key != lastOCRKey || sinceOCR > (call ? 20 : 60)) + if lines.count < 8 || lines.reduce(0, { $0 + $1.count }) < 300 || call, ocrDue { lastOCRKey = key; lastOCR = Date() - let ocr = ocrWindow(pid: app.processIdentifier) + let ocr = ocrWindow(pid: app.processIdentifier, maxWidth: call ? 2400 : 1600) if !ocr.isEmpty { lines = Array(NSOrderedSet(array: lines + ocr)) as? [String] ?? lines; via = "ocr" } } let ms = Int(Date().timeIntervalSince(started) * 1000) @@ -138,8 +176,11 @@ final class ScreenJournal { if seen.count > 20000 { seen = seen.filter { now.timeIntervalSince($0.value) < 3600 } } if !fresh.isEmpty || lines.isEmpty { note("\(bundle): \(lines.count) lines via \(via), \(fresh.count) new (\(ms)ms)") } guard !fresh.isEmpty else { return } - append(["t": ISO8601DateFormatter().string(from: now), "app": app.localizedName ?? bundle, "bundle": bundle, - "window": title, "url": url ?? NSNull(), "via": via, "lines": fresh]) + var entry: [String: Any] = ["t": ISO8601DateFormatter().string(from: now), "app": app.localizedName ?? bundle, "bundle": bundle, + "window": title, "url": url ?? NSNull(), "via": via, "lines": fresh] + if call { entry["call"] = true } + if lastLook { entry["lastLook"] = true } + append(entry) if now.timeIntervalSince(lastPrune) > 24 * 3600 { prune() } } @@ -177,7 +218,7 @@ final class ScreenJournal { /// Capture ONLY the frontmost window of `pid` with the system screencapture tool (same path the app's /// OCR already uses), recognise text on-device, delete the image immediately. Needs Screen Recording /// permission; macOS shows its own prompt once, the user decides there. - private func ocrWindow(pid: pid_t) -> [String] { + private func ocrWindow(pid: pid_t, maxWidth: Int = 1600) -> [String] { guard CGPreflightScreenCaptureAccess() else { if !askedScreenPermission { askedScreenPermission = true; note("OCR needs Screen Recording permission") DispatchQueue.main.async { _ = CGRequestScreenCaptureAccess() } } @@ -196,8 +237,8 @@ final class ScreenJournal { do { try cap.run(); cap.waitUntilExit() } catch { return [] } guard var img = NSImage(contentsOfFile: path)?.cgImage(forProposedRect: nil, context: nil, hints: nil) else { return [] } // Retina windows are ~3k px wide; text stays legible at 1600, and Vision's cost scales with pixels. - if img.width > 1600 { - let w = 1600, h = img.height * 1600 / img.width + if img.width > maxWidth { + let w = maxWidth, h = img.height * maxWidth / img.width if let ctx = CGContext(data: nil, width: w, height: h, bitsPerComponent: 8, bytesPerRow: 0, space: CGColorSpaceCreateDeviceRGB(), bitmapInfo: CGImageAlphaInfo.noneSkipLast.rawValue) { ctx.interpolationQuality = .medium @@ -221,6 +262,15 @@ final class ScreenJournal { private var lastNote = "" private func note(_ s: String) { if s != lastNote { lastNote = s; godLog("journal: " + s) } } + /// Entries produced elsewhere (CallAudio transcripts): masked like screen text, appended on the journal queue. + func record(_ entry: [String: Any]) { + queue.async { + var e = entry // audio / camera have their own switches (Senses.swift) + if let lines = e["lines"] as? [String] { e["lines"] = lines.map { self.mask($0) } } + self.append(e) + } + } + // MARK: storage private func append(_ entry: [String: Any]) { @@ -246,16 +296,18 @@ final class ScreenJournal { } } - private func config() -> (block: Set, blockWords: [String], retentionDays: Int) { - var block = Self.defaultBlock, words = Self.defaultBlockWords, days = 30 + private func config() -> (block: Set, blockWords: [String], retentionDays: Int, calls: Bool, audio: Bool) { + var block = Self.defaultBlock, words = Self.defaultBlockWords, days = 30, calls = true, audio = false let path = (relay as NSString).appendingPathComponent("journal.json") if let d = FileManager.default.contents(atPath: path), let j = try? JSONSerialization.jsonObject(with: d) as? [String: Any] { (j["block"] as? [String])?.forEach { block.insert($0.lowercased()) } (j["blockWords"] as? [String])?.forEach { words.append($0.lowercased()) } if let r = j["retentionDays"] as? Int, r > 0 { days = r } + if let c = j["calls"] as? Bool { calls = c } + if let a = j["audio"] as? Bool { audio = a } } - return (block, words, days) + return (block, words, days, calls, audio) } /// Mask secret-looking tokens before they ever touch disk (API keys, long hex/base64 blobs, card numbers). diff --git a/packages/menubar/Senses.swift b/packages/menubar/Senses.swift new file mode 100644 index 0000000..636fc00 --- /dev/null +++ b/packages/menubar/Senses.swift @@ -0,0 +1,102 @@ +// Senses — one place for what Switchboard is allowed to sense, each switched on its own: +// • screen — the screen journal (ScreenJournal.swift) ~/.relay/journal-on +// • audio — meeting transcripts (CallAudio.swift) journal.json "audio": false | "ask" | "always" +// • camera — presence + enrolled people (CameraPresence.swift) journal.json "camera": true | false +// The notch (on hover), the loose pill and the Journal tab all use SensesPill, so every place shows the same three +// switches. Changes take effect within seconds; each sense re-reads its setting on its own timer. +import SwiftUI + +enum Senses { + static let relay = (NSHomeDirectory() as NSString).appendingPathComponent(".relay") + static var flag: String { (relay as NSString).appendingPathComponent("journal-on") } + static var configPath: String { (relay as NSString).appendingPathComponent("journal.json") } + + static func config() -> [String: Any] { + guard let d = FileManager.default.contents(atPath: configPath), + let j = try? JSONSerialization.jsonObject(with: d) as? [String: Any] else { return [:] } + return j + } + static func set(_ key: String, _ value: Any) { + var j = config(); j[key] = value + if let d = try? JSONSerialization.data(withJSONObject: j, options: [.prettyPrinted]) { + FileManager.default.createFile(atPath: configPath, contents: d, attributes: [.posixPermissions: 0o600]) + } + } + + static var screen: Bool { FileManager.default.fileExists(atPath: flag) } + static func setScreen(_ on: Bool) { + if on { FileManager.default.createFile(atPath: flag, contents: nil) } else { try? FileManager.default.removeItem(atPath: flag) } + } + /// "off" | "ask" | "always" + static var audio: String { + let v = config()["audio"] + if let b = v as? Bool { return b ? "ask" : "off" } + if let s = v as? String, ["ask", "always"].contains(s) { return s } + return "off" + } + static func setAudio(_ mode: String) { set("audio", mode == "off" ? false as Any : mode as Any) } + static var camera: Bool { config()["camera"] as? Bool ?? false } + /// Camera is parked (founder, 2026-09-29): its switch, the viewfinder and People stay hidden unless "cameraLab": true. + static var cameraLab: Bool { config()["cameraLab"] as? Bool ?? false } + static func setCamera(_ on: Bool) { set("camera", on) } +} + +/// Three switches (+ the journal). `compact` = icon-only, for the notch and the loose pill. +struct SensesPill: View { + var compact = true + @State private var screen = Senses.screen + @State private var audio = Senses.audio + @State private var camera = Senses.camera + + var body: some View { + HStack(spacing: compact ? 5 : 8) { + SenseToggle(symbol: screen ? "eye" : "eye.slash", label: "Screen", on: screen, color: .lime, compact: compact, + help: screen ? "Screen journal is on. Click to stop reading the screen." : "Screen journal is off. Click to start.") { + screen.toggle(); Senses.setScreen(screen) + } + SenseToggle(symbol: audio == "off" ? "mic.slash" : "waveform", label: audio == "always" ? "Meetings: always" : (audio == "ask" ? "Meetings: ask" : "Meetings: off"), + on: audio != "off", color: .senseAudio, compact: compact, + help: "Meeting audio: \(audio). Click to cycle off → ask first → always.") { + audio = audio == "off" ? "ask" : (audio == "ask" ? "always" : "off"); Senses.setAudio(audio) + } + if Senses.cameraLab { + SenseToggle(symbol: camera ? "camera" : "video.slash", label: "Camera", on: camera, color: .senseCamera, compact: compact, + help: camera ? "Camera presence is on. Click to turn the camera off." : "Camera is off. Click to turn on presence (at desk / away / people you enrolled).") { + camera.toggle(); Senses.setCamera(camera) + } + // The viewfinder: see who's in view and tag people (CameraPanel.swift). + SenseToggle(symbol: "person.crop.rectangle", label: "Tag people", on: true, color: .senseCamera, compact: compact, + help: "Open the camera viewfinder to see who's in view and tag people (with their OK).") { + CameraPanelController.shared.show() + } + } + } + .onAppear { screen = Senses.screen; audio = Senses.audio; camera = Senses.camera } + } +} + +private struct SenseToggle: View { + let symbol: String + let label: String + let on: Bool + let color: Color + let compact: Bool + let help: String + let action: () -> Void + @State private var over = false + var body: some View { + HStack(spacing: 5) { + Image(systemName: symbol).font(.system(size: compact ? 9 : 11, weight: .bold)) + if !compact { Text(label).font(.hanken(12, on ? .semibold : .regular)) } + } + .foregroundColor(on ? (over ? Color.page : color) : Color.inkFaint) + .frame(minWidth: compact ? 22 : nil, minHeight: compact ? 18 : nil) + .padding(.horizontal, compact ? 0 : 10).padding(.vertical, compact ? 0 : 6) + .background(Capsule().fill(on && over ? color : Color.page)) + .overlay(Capsule().stroke(on ? color.opacity(0.8) : Color.edge, lineWidth: 0.8)) + .contentShape(Capsule()) + .onHover { over = $0 } + .onTapGesture(perform: action) + .help(help) + } +} diff --git a/packages/menubar/build.sh b/packages/menubar/build.sh index 3bf25f7..b06e43b 100755 --- a/packages/menubar/build.sh +++ b/packages/menubar/build.sh @@ -5,7 +5,7 @@ cd "$(dirname "$0")" echo "[menubar] compiling…" mkdir -p build -swiftc -O -o build/Relay main.swift RelayMenuBar.swift CursorGuide.swift TeamSection.swift TeamCursorsOverlay.swift IgnitionOverlay.swift DictationScratch.swift WhiteboardPanel.swift NotchTray.swift NotchLauncherView.swift LauncherRouting.swift ConsentRouting.swift GodWidgetKit.swift GodWebWindow.swift StoreFrontView.swift HtmlCapability.swift SkillRunner.swift AmbientSensor.swift AmbientCanvas.swift ScreenJournal.swift LooseNotch.swift OSShellView.swift OSSurfaceWorkspace.swift OSSurfaceAutomate.swift OSSurfaceKnowledge.swift OSSurfaceDo.swift -framework AppKit -framework SwiftUI -framework WebKit -framework ApplicationServices -framework CoreServices +swiftc -O -o build/Relay main.swift RelayMenuBar.swift CursorGuide.swift TeamSection.swift TeamCursorsOverlay.swift IgnitionOverlay.swift DictationScratch.swift WhiteboardPanel.swift NotchTray.swift NotchLauncherView.swift LauncherRouting.swift ConsentRouting.swift GodWidgetKit.swift GodWebWindow.swift StoreFrontView.swift HtmlCapability.swift SkillRunner.swift AmbientSensor.swift AmbientCanvas.swift ScreenJournal.swift LooseNotch.swift CallAudio.swift CallCaptions.swift CameraPresence.swift Senses.swift CameraPanel.swift OSShellView.swift OSSurfaceWorkspace.swift OSSurfaceAutomate.swift OSSurfaceKnowledge.swift OSSurfaceDo.swift OSSurfaceJournal.swift -framework AppKit -framework SwiftUI -framework WebKit -framework ApplicationServices -framework CoreServices APP="Switchboard.app" rm -rf "$APP" From 3cc96cd2536639ed6d428823b866d8eaa793adeb Mon Sep 17 00:00:00 2001 From: sameeeeeeep <191771413+sameeeeeeep@users.noreply.github.com> Date: Tue, 29 Sep 2026 18:35:04 +0530 Subject: [PATCH 4/4] Fix 0.4.1 staging: eye colour/gaze declarations belong on NotchField, not inside eyeLamp Staging hunks without context placed them 30 lines low; the release build caught it. Co-Authored-By: Claude Opus 5.5 --- packages/menubar/RelayMenuBar.swift | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/packages/menubar/RelayMenuBar.swift b/packages/menubar/RelayMenuBar.swift index a0909d1..501b0b9 100644 --- a/packages/menubar/RelayMenuBar.swift +++ b/packages/menubar/RelayMenuBar.swift @@ -789,10 +789,10 @@ final class Model: ObservableObject { @Published var userName: String = "" // what God calls you (~/.relay/profile.json → name) @Published var economy = false // prefer a cheaper/faster model to spend fewer tokens @Published var journalOn = false // screen journal recording (~/.relay/journal-on) → eyes on the notch - @Published var creatureLoose = false // the Loose Notch pill is off the notch (LooseNotch.swift) → the eyes go with it - @Published var regionSelect = false // ⌃⌃ lets you drag a screen region → only that is sent @Published var listening = false // CallAudio is transcribing a call → the eyes turn amber @Published var cameraOn = false // CameraPresence is watching → the eyes turn coral + @Published var creatureLoose = false // the Loose Notch pill is off the notch (LooseNotch.swift) → the eyes go with it + @Published var regionSelect = false // ⌃⌃ lets you drag a screen region → only that is sent @Published var defaultShare = false // ⌃⌃ auto-shares the whole screen (fn+click then TOGGLES it off) @Published var modelProviders: [ModelProvider] = [] @Published var originModels: [String: String] = [:] // origin → its current model (grants.json modelOverride) @@ -2398,6 +2398,9 @@ struct NotchField: View { /// Screen journal recording → the field's OWN lamps form two eyes (no overlay): lamps inside the eye /// shapes glow full, the rest of the field dims so they read. Blink + glance ride the same clock. var eyes: Bool = false + var eyeColor: Color? = nil // nil = the accent; amber while listening to a call + /// Screen point of this field's centre; when set, the eyes look toward the mouse pointer. + var gaze: (() -> CGPoint?)? = nil @Environment(\.accessibilityReduceMotion) private var reduceMotion // One eye on the lamp grid: 4 wide × 3 tall with the corners off (an almond). A blink keeps only the middle row. private static let eyeShape: [(Int, Int)] = [(1, 0), (2, 0), (0, 1), (1, 1), (2, 1), (3, 1), (1, 2), (2, 2)] @@ -2428,9 +2431,6 @@ struct NotchField: View { // Lamp density matches the dictation DotMatrix (founder-approved 2026-08-13, "chunky lamps"): // pitch 6 / dot 3 — bolder + less dense than the old fine field (pitch 4 / dot 1.8). let gap: CGFloat = 6.0, d: CGFloat = 3.0 - var eyeColor: Color? = nil // nil = the accent; amber while listening to a call - /// Screen point of this field's centre; when set, the eyes look toward the mouse pointer. - var gaze: (() -> CGPoint?)? = nil let cols = max(1, Int(size.width / gap)), rows = max(1, Int(size.height / gap)) let ox = (size.width - CGFloat(cols - 1) * gap) / 2, oy = (size.height - CGFloat(rows - 1) * gap) / 2 for c in 0..