Live Mode (voice)

Design: Live Mode — talk to your tmux

Status: v0.1 implemented, multi-provider (this doc tracks what ships). Voice-only surface: a Live button in the header opens a session where you talk in natural language about — and into — every pane at once. The model hears you, sees the live state of all panes, answers out loud, and can type into a pane, as the one and only action it can take. Which model answers is a server-side table; with more than one configured, the Live button becomes a picker (see “Model providers”).

For the supported iPhone workflow, follow Live Mode + notifications setup : Safari or Chrome → Add to Home Screen → launch the installed app. The guide covers microphone and push permissions, screen timeout, lock/background audio interruptions, and recovery.

GPT-Live 1

openbus/gpt_live.py adds OpenAI’s full-duplex voice frontend with managed Responses delegation to gpt-5.6-luna. It reuses the existing pane-context updater, microphone forwarder, tool schemas, and guarded terminal dispatcher. Detailed terminal procedures stay in the backend prompt; the voice prompt covers conversation and delegation.

Configure OPENAI_API_KEY in ~/.config/tmux-rc/openai.env, outside every checkout. Set TMUXRC_LIVE_MODE=1 in the daemon environment (or root .env) and restart the daemon to load both settings. GPT-Live 1 then appears in the picker and is chosen by its label, like every other entry — but it is NOT a TMUXRC_LIVE_MODELS entry and cannot be one: the seam opens a connection and hands it to the shared coroutines, whereas this adapter owns the whole session. So it is routed by label ahead of the table lookup and appended to /api/version’s menu under the same “only if its key is set” rule the table entries follow; the two lists have to agree on what is selectable, or the socket would refuse the one pick the menu just offered. TMUXRC_GPT_LIVE_VOICE chooses the voice (default marin). TMUXRC_GPT_LIVE_BACKEND chooses the Responses model; changing it requires explicit input/cached/output prices (see .env.example). This adapter reaches OpenAI directly — Azure serves the Realtime protocol, which the table’s azure-openai backend uses, but not the Live delegation protocol this one speaks.

The existing browser → daemon WebSocket path remains. GPT-Live negotiates 16 kHz PCM in both directions; audio messages carry their playback rate, and the server requests 40 ms microphone batches. Muting the mobile mic disables the capture track but keeps silent audio flowing, so the Live clock and backend speech continue. There is no VAD commit, response trigger for audio, or invented spoken-turn completion. Transcript timestamps provide display-only gap grouping; overlapping fragments never run tools.

Full pane updates go into backend conversation context. Changed pane digests also arrive as quiet session.thinking.append hints, conservatively bounded below its 500-token limit. These hints never authorize actions. Function calls are collected from completed output items and run only after successful backend completion; every result precedes continuation. A separate worker keeps audio flowing during terminal actions. Duplicate call IDs stop the session rather than repeat keystrokes.

Voice billing uses cumulative usage.seconds at $0.05/min, including silence; backend usage is metered separately, including cached tokens. OTel reports the provider, backend model, voice seconds, and whether final usage was confirmed. Backend completions produce usage snapshots, not voice-turn counts. Stop/disconnect cancels pending local work and drains session.closed for up to five seconds. A lost provider connection ends the session; the adapter does not replay actions or silently start a fresh conversation. Speech interruption alone does not undo an action already sent to a terminal.

Validation: make test; the opt-in, billable uv run python research/live-eval/smoke_gpt_live.py /tmp/live-smoke.wav (live-eval scripts ) synthesizes a fixed utterance and checks speech → delegation → exactly one guarded action → spoken confirmation → final usage. It uses a fake terminal and requires ffmpeg. Phone testing is still needed for echo, interruptions, mobile backgrounding, and perceived latency.

Protocol references: WebSockets , delegation , and session lifecycle .

Why voice, why now

tmux-rc already solved observation (cards, summaries, the live frame stream) and targeted input (the composer types into the selected pane). What it can’t do is span panes conversationally: “what’s the codex pane stuck on?”, “tell claude yes”, “run the tests again in the build pane” — each of those is trivial to say and clumsy to do by tapping a card, reading, switching, typing. A voice loop over the whole session is the natural next interface, and everything it needs already exists in the daemon: per-pane state that is always current (the watcher’s whole job), and key injection (send_keys). Live Mode is a thin conversational layer over those two primitives — it adds no new observation machinery and no new control channel, staying inside the founding “observe the terminal, don’t drive the agent” invariant.

Architecture (two hops, mirrored from a proven implementation)

browser → daemon WebSocket → the model's bidirectional stream (Gemini Live or OpenAI Realtime)

The browser only captures and plays audio; the daemon owns the model session. This is deliberately modeled line-for-line on a battle-tested internal implementation of the same shape (its session wrapper, reconnect loop, tool dispatch, and audio pipeline), because that work already burned down the non-obvious failure modes — see “Traps inherited” below.

Client: getUserMedia mic → AudioContext at 16 kHz → 4096-sample Float32→Int16 PCM frames → base64 over the WebSocket. A gain-0 node keeps the graph alive without echoing the mic to the speakers. Playback of the model’s voice is the reverse: base64 24 kHz PCM chunks → queued AudioBufferSourceNodes.
Daemon (openbus/live.py): accepts the WS, builds the system prompt from live watcher state, opens the chosen model through the provider seam (openbus/live_providers.py), then runs two concurrent coroutines — forward-audio-up and receive-events-down — plus a context-updater (below). Session drops reconnect with exponential backoff.
Model: whichever entry of the server’s table the user picked; the default table is gemini-live-2.5-flash-native-audio on Vertex (us-central1). Always AUDIO response modality with input+output transcription on, so the UI can show a text transcript of both sides whoever is answering.
The Gemini Live client is built separately from the classifier’s cached client: Live needs us-central1 (not the classifier’s global) and must NOT inherit the classifier’s 20s per-request timeout — a live session is one long-lived stream.

Two transport lessons from first deployment, both invisible until a real WebSocket is in the path (everything else in the app is request/response):

The access proxy must pass WebSocket upgrades. Any HTTP-aware layer in front of the daemon (auth proxy, tunnel, buffering reverse proxy) that serializes requests as request→response will silently break wss:// — the daemon sees a plain GET on a websocket-only route and 404s. If your fronting layer can’t stream upgrades, fix it there rather than contorting Live Mode into chunked-POST/long-poll transport: voice needs the latency, and the workaround code would outlive the proxy limitation.
Broken-IPv6 networks starve the Vertex handshake. The Live endpoint resolves to AAAA records first; a network with an advertised-but-dead v6 route (common behind home routers) eats the entire open timeout walking them serially, so the session loops connecting → reconnecting while plain HTTPS (with its fast fallback) looks healthy. The daemon sorts IPv4 first in getaddrinfo at startup — harmless where v6 works, unbreaks all egress where it doesn’t.

How the model sees pane state (the part that makes it “talking to tmux”)

The watcher already keeps every pane’s state current — that is the daemon’s core loop. Live Mode simply reads it; there is no separate polling and no LLM call until you speak.

At connect: the system prompt embeds a full snapshot — every pane’s id, label, tool, activity, headline, idle summary, pending question, and the tail of its current screen text. This is the same data the phone’s cards render, sourced from watcher.digest() + the latest snapshot per pane.
While the session runs: a context-updater coroutine waits on the watcher’s state_version (the same change signal the /api/state long-poll uses — one source of truth for “something changed”) and, throttled to at most one update every few seconds, injects a compact [tmux update] message into the session with turn_complete=False. That flag is the key mechanism: the content lands in the model’s context without prompting a response turn, so the model never chatters at a state change — but when you next speak, its view is already current. This is exactly the “state is just always up to date; a turn happens when I talk” model.
After the model types: the tool handler forces a daemon reparse of the target pane and, after a short settle, sends that pane’s fresh screen tail as another ambient update — so the model sees what its own keystrokes did without us echoing anything through the tool response (see traps).

Updates are digest-sized (labels, activities, headlines, one-line summaries, pending questions), not full screens — full screens ride only in the connect snapshot and in the post-type refresh of the acted-on pane. That keeps a long-running session’s token drip small while keeping answers accurate; if it proves too coarse, the escalation path is a screen-tail in updates for the active pane only.

Two tools: `type_in_pane` and `press_key`

type_in_pane(pane_id, text, press_enter=true)   # type a string
press_key(pane_id, key)                          # key ∈ {Escape, Enter, Up, Down,
                                                 #        Left, Right, Tab, C-c, C-d}

type_in_pane types exactly text into a pane via the daemon’s existing send_keys (literal mode), optionally submitting with Enter. press_key sends one named control key non-literally with no auto-Enter — the actions that aren’t text: cancel (Escape), interrupt (C-c), menu navigation (arrows), accept/continue (Enter), complete (Tab), EOF (C-d). Both go through the same send_keys primitive every other tmux-rc input path uses.

Started as a single type_in_pane and that was a mistake in practice: asked to “stop Claude” or “press escape,” the model narrated pressing a key it had no way to send — the tool couldn’t express anything but literal text. The fix is a second, deliberately narrow verb rather than overloading the first: two unambiguous actions (type a string / press a key) the model chooses between, and press_key’s whitelist means it can only send keys that make sense for a terminal UI, never an arbitrary chord we didn’t vet. The prompt teaches the vocabulary generically (Escape cancels, C-c interrupts, arrows+Enter pick a menu item) so it works for claude/codex/gemini/opencode/omp/shell alike — no per-vendor cheatsheet to go stale, matching the “observe the terminal, don’t model the agent” principle.

The shared handler:

rejects calls with a missing/unknown pane_id, unexpected extra args, non-bool press_enter, empty text, or a non-whitelisted key (the model occasionally echoes a prior FunctionResponse back as a new call — inherited guard);
executes, then returns a terse FunctionResponse (done in <label>) carrying the original call id — never the resulting screen;
notifies the browser ({type:"typed", ...}) so every action the voice takes is visibly logged in the overlay — you always see what it did and where;
forces a reparse and schedules the post-action context refresh described above.

Deliberately absent: pane switching (the model doesn’t need focus to act — send_keys targets any pane), pane creation (“open claude in a new pane in ~/src/x” is the obvious v0.2; POST /api/windows already opens configured launchers), kill/destroy (destructive, needs the risk tiers in the control plane ), and any read tool (state is pushed, not pulled).

Prompting strategy (where the quality lives)

The reference implementation’s conversational quality came from prompt iteration, not architecture, and its lessons transfer even though its use case was the polar opposite (a passive meeting listener that errs toward silence vs. our active command executor). What carried over, adapted:

Terse, imperative, behavior-first. No persona preamble. Every sentence is an operating rule. The reference prompt is ~15 lines; ours stays in that class.
Voice discipline. The model speaks; so: one-or-two spoken sentences, no markdown, no lists, no code blocks read aloud. Without this, audio models recite formatting.
Noise rules survive verbatim. A live mic picks up keyboard clatter, other people, fragments. The reference’s “ignore background, don’t respond to fragments, don’t invent questions, never act twice on the same request” rules apply unchanged — they’re about the microphone, not the product.
The tool-contract lesson, inverted. The reference had to teach “the query system CANNOT see the screen — you must translate visual references into explicit terms.” Our version of that hard-won line: “type_in_pane types EXACTLY your text into a real terminal, as if the user typed it — you are typing, not chatting; say what you’re typing and into which pane as you do it.” The model must understand it is operating a physical-ish effector with a precise contract, not emitting conversation.
Where the reference says “err toward silence,” we split the rule in two. Answering questions is low-stakes — be helpful, answer from state, admit when the state doesn’t show something (never invent terminal output). Typing is high-stakes — only on a clear user request, exact text only, and a hard rule against destructive commands (rm -rf, force-push, kill) unless the user spelled them out. The passive/active split lives inside the prompt rather than in the architecture.
Ambient context is labeled and fenced. [tmux update] messages are declared to be state refreshes, not the user speaking: “never respond to them; treat them as current truth.” Belt (the turn_complete=False mechanism) and suspenders (the prompt rule).
Per-pane audience awareness. Agent panes (claude/codex/gemini/opencode/omp) receive natural language — the agent parses it. Shell panes receive exact commands. Panes showing a numbered menu can be answered with the option the prompt expects. This routing knowledge lives in the prompt because the pane state already carries each pane’s tool.

Traps inherited (paid for elsewhere, honored here)

Never feed tool results back as conversation. Returning rich results in the FunctionResponse induced feedback/echo loops in the reference (the model re-emitting responses as new tool calls). We return a terse status and deliver the actual outcome as ambient state instead.
Carry fc.id on every FunctionResponse — omitting it wedges the session.
Reject malformed/echoed tool calls (extra args, empty payload) instead of executing them.
Model/region/modality must match exactly — a live model id on the wrong region fails opaquely. Pinned: gemini-live-2.5-flash-native-audio @ us-central1, AUDIO.
Proactive audio (proactivity.proactive_audio=true) lets the model decide not to respond — required for the noise rules to actually work rather than the model answering every sound. It is a per-entry flag, not a constant: Gemini 3.1 Flash Live rejects the setup field outright (found the hard way in an earlier prototype).

Alternatives considered

Text chat first, voice later. Rejected for this feature (text chat now arrives as a front end in control session and routing ): the point here is specifically the hands-free, cross-pane conversation; a text box duplicates the composer we already have.
Browser talks to Gemini directly (no daemon hop). Rejected: the tool must execute send_keys on the daemon anyway, credentials live server-side, and the daemon is where pane state lives. Two hops is the only shape that doesn’t scatter authority.
A read_pane tool instead of pushed context. Pull keeps tokens minimal, but adds a round-trip before every answer and a second tool to prompt around. Push matches how the rest of tmux-rc works (the watcher maintains truth; consumers read it) and makes answers instant. Revisit only if ambient updates prove too expensive.
Reusing the classifier’s Vertex client. Rejected — see architecture: wrong region default, and its per-request timeout (a deliberate anti-wedge guard for one-shot parse calls) would sever a long-lived live stream.
Full screens in every ambient update. Simplest mental model, unbounded token drip. Digest-level updates + targeted screen refreshes cover the same questions at a fraction of the cost.
An approval gate before typing. The control plane calls for risk-tiered confirmation before acting. For v0.1 voice we deliberately run without one: the session is explicitly started by the user, every action is spoken aloud and logged in the overlay, and the prompt constrains typing to explicit requests. If real use shows misfires, the gate design is already written.

Cost metering

Voice is billed unlike anything else in the app: mostly AUDIO tokens, at roughly 30× the flash-lite parser’s text rate, so a few minutes of talking can outspend an hour of parsing. It gets its own accounting rather than being folded into the parser’s:

Per-modality, per-model pricing. A session keeps text and audio in/out separate (_LiveUsage) — a single blended price would be wildly off since audio-out alone is ~24× text-in — and prices them with the rate card on the model’s table entry. The card is per entry because one global card would be wrong the moment a second model is on the menu, and an OpenAI session at ten times Gemini’s audio rate must show up as such in the status bar, not be quietly priced as Gemini. A missing rate defaults to Gemini 2.5’s, so a new entry is never a silent zero.
Cached input has its own two rates. Realtime re-bills the entire conversation as input on every response, and the server caches most of it at a 97-99% discount — so in any session longer than a few turns, cached tokens are the majority of input. Pricing them at list (the first cut did, calling it a “slight overcount”) overstates a long session severalfold, which defeats the point of a side-by-side cost comparison. The providers return token counts only, never a price, so the split has to be carried through: the cached counts leave the text/audio input buckets and are billed at the card’s text_cached/audio_cached rates. Those default to the entry’s OWN uncached rate rather than Gemini’s: a model with no published cache discount gets none, so the default is never a silent undercount either.
Providers report connection-cumulative totals, so the meter overwrites. Gemini sends running session totals on every message; the OpenAI adapter sums its per-response usage into the same shape (each Realtime response re-bills the whole context as input — that is the real cost shape of the API, and summing is what makes it visible). Either way the last event carries the totals and the accumulator overwrites rather than sums.
OTel. A tmux-rc.live record per voice turn plus a final cumulative row at session end (emit_live_turn), keyed by the same per-page session UUID as the live-view watch-time rounds — so a query joins voice spend to screen time. Privacy matches emit_parse: numbers + structure always; the actual transcript (what was said and typed) only under TMUXRC_QSDEBUG, voice being at least as sensitive as pane text.
Status bar. Session cost folds into llm.usage_totals() at session end as a distinct live bucket, so the top-bar $cost is total (parser + voice) with the tooltip splitting the two; a 🎙<n> chip shows how many voice sessions ran.

Model providers (side-by-side)

The 2.5 voice sounded dated next to current consumer assistants, and the only honest way to judge a voice model is to talk to it from the phone about real panes — a text harness (research/live-eval/ ) shows every candidate passing the same tool-calling cases at sub-second latency, so reasoning is not what separates them; naturalness, barge-in and turn-taking are, and no harness sees those. Hence a picker on the real surface, backed by a small amount of deliberate structure:

The model list is server-owned, and the browser only ever names a label. Same rule as the launcher menu, for the same reason: the WebSocket must never accept a raw model id, backend or credential from a client. TMUXRC_LIVE_MODELS holds the table (inline JSON or a path — the launchers shape, through one shared helper); /api/version lists the offered labels with a provider/cost hint; ?model=<label> on the session socket is resolved against that list and anything not offered is refused with a policy-violation close rather than defaulted — a picker that quietly answered with a different model would make every side-by-side impression a lie.
Backend is per entry, because auth is. Gemini 2.5 stays on Vertex with the durable service-account story (durable-vertex-auth.md); Gemini 3.x Live exists only on the AI Studio API, so it needs an API key; OpenAI Realtime needs its own key, or an Azure resource where the “model” is a deployment name. An entry whose keys are absent stays configured but unoffered, so the picker appears the moment a key lands with no config edit. Keys live in one mode-600 file outside every checkout (~/.config/tmux-rc/openai.env), never in a worktree’s .env.
Capability flags are per entry too. 3.1 Flash Live rejects the proactivity field that 2.5 depends on; the OpenAI voice name means nothing to Gemini. Anything a backend might reject rides on the entry, not in shared code.
A session protocol, not if provider: branches. live.py’s coroutines speak four verbs and one event stream (send audio, send context, send tool result, events out); each provider is an adapter behind them. The seven functions that used to touch the Gemini SDK moved; nothing else changed. The tools are declared once as plain JSON Schema, which both SDKs accept as-is.
What “context without a reply” means on each wire. Gemini: a client-content turn with turn_complete=False. OpenAI: a system-role conversation item with no response.create — system rather than user role so a state refresh can never be mistaken for the user speaking (an earlier prototype saw user-role injections re-trigger actions). OpenAI also does not continue a turn after a tool result on its own; the adapter asks for the follow-up, deferring until the current response has finished because asking mid-response is a protocol error.
Audio is normalised server-side. The browser captures 16 kHz and plays 24 kHz; Realtime is 24 kHz both ways. The adapter resamples input (linear, 2→3) so web/ has one audio pipeline and never learns which provider it is talking to.
Barge-in is a first-class event. Both providers signal that the user spoke over the model (Gemini interrupted, Realtime speech_started); the browser cuts its queued playback so the reply stops with the model instead of talking through the user. It matters here precisely because interruption handling is one of the things being compared.
A model that cannot be reached says so. A wrong Azure deployment name or a rejected key fails at the handshake with a message naming the deployment and host; that error is not retried (the reconnect loop cannot fix config) and is shown verbatim, unlike in-session failures which stay generic.
Picker UX. The Live button is the picker: with one offered model it starts a session exactly as before; with several it opens a bottom sheet — one thumb-sized row per model with the provider and audio rate as a hint, the remembered choice ticked — and tapping a row starts the session at once. While connected the pill’s tag shows the model’s label, so during a comparison you always know which voice you are hearing. A header dropdown was rejected: the top bar has no room on a phone, and the choice is made mid-conversation with a thumb.

Rejected: routing OpenAI through Bedrock (it carries no realtime models) and gating on the harness alone (see above). The cost argument against OpenAI stands — ten times Gemini’s audio rate — which is why the status-bar fold-in must use each session’s own card: that number is where the comparison will be decided.

Text mode (prototype)

This is a first cut of the composer described in live-conversation-history.md. It keeps the same session, tools, fleet snapshot and audit, with typed turns in place of speech. A typed turn is a new adapter verb, send_text, deliberately separate from send_context. Context must never prompt a reply, and a typed turn must always get one. The daemon echoes the turn back as the user’s transcript. The bubble therefore appears only once the daemon has the turn, never as an optimistic local copy, and both clients render typing exactly like speech.

Text runs on chat models, not the voice pipeline. The first prototype reused the realtime seam, which meant the only models that could reply in text at all were GPT Realtime 2.1 and its mini, since Gemini Live’s native-audio models refuse text output and GPT-Live only speaks. Those are small models tuned for latency, and in use they were noticeably worse at choosing panes and reading results than an ordinary chat model. A typed request has no latency budget that forces that trade, so text sessions now run on Gemini Flash (through the classifier’s existing Vertex client and credentials) or Claude Sonnet. live_chat.py implements the same session verbs as the voice adapters over a plain request/response tool loop: send the conversation, run any tool calls, append the answers, repeat until a reply has none. Because it sits behind the same seam, every call still goes through _handle_tool_call, so the audit and the consent gate below apply unchanged and the browser cannot tell which kind of backend answered.

A few differences follow from the wire. A chat API has no reply-less channel, so pane updates are held (the last few, since they supersede each other) and sent in front of the next typed turn rather than injected as they happen. A failed request costs only that turn: reconnecting, as voice does, would throw away the conversation, which here lives in the daemon rather than in a server-side session. A rejected key, missing access or unknown model still ends the session with the reason, because no retry fixes config. A turn is capped at a handful of model requests, so a model that keeps calling tools instead of answering stops rather than spending without bound. A turn that ends that way, or fails, is closed in the conversation with the same note the user saw; left open, the next typed turn would send the abandoned request (or tool chain) again and the model could act on it. The conversation keeps the last twenty turns, dropped whole from the front so every tool call keeps its result: each request resends everything, so an unbounded session would cost more per turn and eventually overflow the context window.

Flash sometimes answers a tool result with nothing at all, after announcing the action before calling the tool. The prompt now says to act first and report after, and a silent response after tool activity is asked once for a one-sentence outcome. If it stays silent the turn ends without a placeholder, because the approved card and the typed row already show what happened and a “no reply” line under them reads as a failure. The placeholder is kept for a turn with no tool activity, where it is the only sign the request was heard.

The mode picks the list: chat entries appear only under Text and everything else only under Voice, and the socket refuses the other mode’s model. Chat entries live in the same TMUXRC_LIVE_MODELS table (backends vertex-chat and anthropic). A table that names none of them gets Gemini 3 Flash and Claude Sonnet 5.5 appended, so a table written for voice never leaves Text empty. A keyless entry is listed greyed with the credential it needs, as launchers are, instead of vanishing: the user configured it and should see why it can’t run. For the same reason the Live button stays visible when every entry is keyless; hiding it would leave no place to read those reasons. We considered keeping the realtime models as a Text option and rejected it; a choice nobody should make is clutter.

Pane-changing tools wait for a tap. When the assistant answered a typed “What’s going on”, it typed a relayed question into a live agent that nobody had asked it to touch. In a text session, type_in_pane, press_key and resume_session are therefore proposed rather than run. The client shows what the call would do, and only Send executes it. Cancel answers the model that the user declined, and the audit records either answer. The gate sits in _handle_tool_call, the one choke point every Live tool call already passes through, so no tool can skip it. find_sessions only reads, so it runs at once. This is the control plane’s risk-tier idea (agentic-control-plane.md) applied at the smallest scale. Voice keeps acting directly, because a tap would end hands-free use and the spoken “Typing X into Y” announcement is voice’s confirmation.

A card outlives its connection. Cards used to expire the moment the socket dropped, because the daemon’s half of a proposal lived in the coroutine waiting on the tap. On a phone that is every lock, every app switch iOS suspends, and the tunnel relay’s hourly cap, so a card read, considered and then tapped a few minutes later said “Expired” with nothing about it having aged. Now a card whose connection drops is parked, keyed by the user and the page’s chat session id, with what Send needs: the call, the pane process the card named, and the session’s images. The phone keeps the card open, a tap reconnects at once, and the answer runs on the new connection. The model that asked went with the old connection (a reconnect starts a fresh one), so the outcome is shown and audited but not told to it, and a new message supersedes parked cards as it does live ones. A tap is resent on each new connection until answered, since a socket can die with it or its answer in flight, so the daemon also remembers each answer for as long and repeats it rather than running the card twice or calling it expired. The untapped cards are asked after on each new connection too, since a new message may have superseded one with the answer lost the same way. Expiry is real now: thirty minutes parked, a daemon restart, or ending the chat (ended while offline, the next chat’s first connection clears them). Keeping the whole session (model, queue, tool calls) alive across a reconnect would also save the model’s history, but it would turn a per-socket task tree into a detachable one, far more than the card needs.

Two things have to hold for that to survive a real outage rather than a lock. The phone must not give up first: its usual five reconnect tries are over in about thirty seconds, and the tunnel takes a minute to come back from its hourly drop, so while a card waits the chat keeps trying at a capped backoff, and tries at once when the phone comes back online or into view. And a reconnect must find the card parked: a phone that saw its socket die can be back before the daemon notices the old, half-open one at its ping timeout, with the card still held there. So a new connection for a chat takes over the one it replaces, ending that one’s waits exactly as a drop would, and the cards park before the new socket can deliver its first frame.

Taking the user to a window. Asked to “go back to window 20”, the assistant could only say the window was ready and ask what to do there: it could talk about panes but not show one. open_pane closes that gap by putting an Open button in the log, which goes where tapping the pane’s row goes (on a phone it also minimizes the sheet). It changes only the view, so it skips the consent card, and it is a button rather than an automatic jump because the reply can land while the user is typing their next message, and a view that changes under them would lose their place. Voice gets the same tool and button. The one jump is a resume the user just tapped Send on: that tap asked to go there, so the daemon sends the same message marked auto and the client clicks the button it adds, which stays for getting back. The result tells the model the user is already there, so it offers no second button.

The model names the window and the daemon finds it. Left to pick a pane id itself, the model twice opened the wrong window: “take me to the team inbox merge” (a voice slip for “pane”) went to past-session search, whose top hits matched all three words, and opened the pane running one of them rather than the window titled “team inbox”; “the acme linux OMP session” read the tool column and chose an idle omp pane. So open_pane takes the user’s words and scores each live pane by the words its title, label, tool and working directory’s name hold whole (two adjacent words may also match run together, for “acmelinux”); the directory counts because a repo name is how people say “the press kit session”, and only one of three windows in that repo happened to carry it in its title; a “window N” first narrows to panes with that number, so a 25 in some other title never competes. A pane opens only when it alone tops the scores with most of the words; otherwise the closest few each get a button and the model is told to say how they differ (left to offer them itself, it named two windows in its reply and buttoned neither), or it gets “no match” and a pointer to find_sessions. Buttons go inside the reply’s bubble, under its text: a separate card read as a second message about the same thing. A deterministic matcher is testable and cannot be talked out of an exact title; prompt rules for the same ranking would be relitigated on every model change. Typing still takes a pane id: a wrong Open costs a tap back, a wrong prompt costs work, and the id is checked on a card or announced aloud.

A consent card that targets a pane carries the same button, because the card is where a wrong target gets caught: asked for “the acme linux OMP session”, the model picked an omp pane idle for two days over the window titled “acmelinux codex installation”, and the card named it only by its tmux address, so nothing on it said which window that was. The card now names the pane as the list row does (title, tool and session chip, “example-app · codex in example-app”), and Open lets the user look before answering. It used to lead with the window number and append the tmux address, but the phone shows the number only in a row’s small print and the address nowhere, so leading with them read as jargon on the one surface where the user must recognise the target; the address now goes only to the audit record. The window number comes back only to tell two otherwise identical panes apart, since nothing else on their rows differs. Opening does not answer the card: it stays pending on the chat bubble until Send or Cancel, since looking is the point and a card that resolved on navigation would force a choice before the look.

A reconnect hands the conversation back. The conversation lives in the daemon, so a restart (a deploy) dropped it while the client still showed every turn. The user had been asked “want me to reopen it?” and answered “yes” across the restart; the new session, with no history, matched that “yes” to a different window waiting on a yes/no prompt, the user approved a card they read as the reopen, and the model then described work the pane was “now” doing that it had never seen. Fixes, one per failure:

The client resends what its log shows on every connect that has one, and the daemon puts it in front of the next turn as quoted text, framed as restored and unverified. The alternative, persisting each session’s history server-side keyed by actor and session, would write conversation text (which the daemon treats as speech, journaled only under QSDEBUG) to disk and add storage, expiry and cleanup, all to rebuild what the client already holds. The client’s copy is also exactly what the user is answering. It cannot skip consent: it is text in a user turn, never a tool call, result or approval (those exist only as futures in the live process), and anything in it the user could just as well have typed. It is bounded like a turn, keeping the tail. The log row saying it was handed back is the visible marker.
What a bare “yes” agrees to is the chat model’s reading, not a word list’s. A list of agreement words and a question-mark test once refused such turns in code; they judged meaning with rules (every new phrasing was a new pattern) and were removed. The consent card is the safeguard that stays: nothing reaches a pane before the user reads what would be sent and taps Send.
A delivered send answers sent with a note that the pane’s reaction is unseen, instead of done, which the model took as the work being done; the nudge after a quiet tool turn asks what it did, not the “outcome”.
A card carries a warning line when its turn followed a restore, where the user’s intent and the model’s reading most easily part.
A turn ends with the pane the conversation is on: the one the last action reached. In one chat, “ask the agent” right after acting on a pane went to a different pane whose screen shared a word with the request; the history held the answer, but the model weighed a loud screen over it. A fact in the turn itself settles that without examples in the prompt. It goes last, after any screen text, and pane text is flattened to one line, so a screen cannot forge it. It is kept as the pane’s id and process, and rendered when the turn is built, so a closed pane, or a new one reusing its id, drops the line rather than steering the model at a stranger. Only actions move it: a declined card or an open-pane button reached nothing. A queued message keeps the pane shown when it was typed. Inferring the pane from names in the replies, and refusing sends to any other pane in code, were tried and dropped: both judged meaning with word matching, which is the model’s job.
A pending question that is a numbered menu lists its rows in the pane state, and the prompt says a menu takes the one key that picks a row, with no Enter: typed text plus Enter picks the highlighted row, usually “1. Yes”, and an Enter after a digit lands on the next screen. A code gate on keys and rows was tried instead and dropped for the same reason as above; the consent card still shows exactly what would be typed. It does not name the row a digit picks: that label is only honest if the approval is also bound to the question it was read from, as a push answer is, or a menu that moved on while the card waited would take the digit as a different choice than the one shown.

A text session’s prompt labels relayed messages “(via text)” rather than “(via voice)”, and it tells the model its actions are approved first. The mode is fixed when the session starts. Switching mid-conversation, as the history doc asks, now means moving a conversation between a chat model and a voice session, so it waits until the prototype has been used. The composer also appears in voice sessions, where it is handy for pane names nobody can pronounce.

Chat entry

Typing to the assistant used to start at the mic button, then a Voice/Text picker, then a model list. That is three taps and a microphone icon for something that needs no microphone, and the picker’s remembered mode meant a voice user who typed once got Text the next time. The header now has a chat button next to the mic on both clients. It starts a text session on the chat model used last (else the first) without any picker and never asks for the mic; a small switcher inside Chat changes the model, which starts a fresh session because the new model has none of the old history. The mic button is voice only, so the Voice/Text toggle and its remembered mode are gone. If a conversation is already running, in either mode, either button brings that one back instead of starting another. The chat button hides when no chat model has its key, rather than opening onto a refusal.

Naming. The typed experience is “Chat” in every title, button and label; “Live Mode” names only a voice session. Users read “Live” as “it is listening”, which a typed chat never is. Code identifiers keep their live/lm names: renaming them would churn every file this touches for no behavior.

Minimize, not close. A text conversation is most useful next to the pane it is about, and the sheet covers that pane. Minimizing hands the space back and leaves a floating bubble. The session keeps running. The bubble carries a dot for anything said since and a count of consent cards still waiting, because a card nobody sees holds the model’s turn open indefinitely. Restoring keeps the draft and the transcript’s scroll position. Ending the conversation stays an explicit button, so a stray tap on the backdrop never throws one away.

A waiting card notifies. The bubble only helps someone looking at the page. A card left waiting while the sheet is minimized, the app is in the background or the phone is locked sends a “Chat needs you” push through the same subscriptions as a pane that needs you. The client tells the daemon over the chat socket whenever the sheet goes in or out of view, so a card the user is reading stays quiet. A card must wait unseen for the same settle time as a pane’s push, so minimizing for a moment does not buzz, and it pushes at most once. A dropped socket counts as out of view, and the wait belongs to the card rather than to the connection that showed it: locking a phone drops the socket and parks the card, and that is the moment a push matters most. Whether a parked card is in view again is the reconnected chat’s word, including a card it took over from a half-open socket: the takeover ends the old connection’s wait as a drop does, not as an answer, so the push still comes. The notification has no answer buttons: approving a pane action should happen where the card shows exactly what will be typed, not from a lock-screen line. Tapping it brings the chat back in the open page instead of navigating there, since a navigation would reload the page and end the conversation that holds the card. A minimized chat comes back too, scrolled to the card. The app window is focused before it is asked, not after its answer: iOS may refuse a focus that is no longer tied to the tap, and a suspended page cannot answer until it is woken. Only with several app windows open does the worker ask first, so that only the one holding the card is brought forward. The message alone proved not enough on an iPhone: a tap opened the app with the chat still minimized, most likely because iOS loses a message posted to a page it is still waking. So the worker also records the tap, before focusing, in Cache Storage, which both sides can read, and the page pulls it whenever it comes back (visible, shown, focused, booted) and again a moment later, since iOS may show the page before the worker has run. A record older than a minute or naming another tab’s card is ignored, and one acted on is deleted, so a later return never reopens a chat minimized since. Bringing a chat back twice is harmless, which is what lets both paths stay. A page iOS killed meanwhile has no chat to bring back, since the conversation lived in it, so its boot does nothing with the record.

Docked on a wide screen. At the wide layout’s breakpoint (1100px) the same panel opens non-modally as a column right of the work area, so nothing is dimmed and the pane stays usable while you talk about it. It is the same element and session, not a second chat: crossing the breakpoint just reopens it the other way, and a phone keeps the sheet. The column is an implicit third grid track, so the pane or dashboard beside it narrows and the pane’s Auto layout re-picks from the width left over: terminal beside the overview when that is still 900px or more (a 1920px window), the overview stacked above the terminal below that (1280 and 1440). Rejected: a tab strip sharing the overview’s column, which would hide the overview behind a click exactly when the window has room for both. Docked, the chat button toggles the panel, and Escape closes it only from inside, since the rest of the page is live and Escape there belongs to it.

Pasted images. The composer is the pane composer’s own editor (m/composer.js), shared through web/live-chat.js by both clients, so paste and the attach button behave the same and accept the same types and size (PNG, JPEG, WebP or GIF under 20 MB). The daemon checks the types against the pane paste’s table, but its size limit is Chat’s own: at most four images a turn and 8 MB together. Unlike a pane paste, a chat image is resent with every later request while its turn is kept, so the per-turn budget is what bounds a session’s history. A voice session takes them too, where its model can see them: each session says so (images), and the daemon refuses a turn it cannot carry with the actual cause — a voice model without images, too many, too large, wrong type — because a single “limits” message sent a user hunting for a size problem on a small screenshot. Realtime takes input_image in the user item. GPT-Live’s voice frontend takes no images, but its Responses backend does, and typed turns already go there. Gemini Live on Vertex takes them as reply-less client content ahead of the realtime text; as realtime video frames they race the text, which 2.5 answered before seeing the frame. 3.x Live on AI Studio accepts client content only as initial history, so it stays off there. The client re-encodes each image as a JPEG no longer than 1568 px on its long edge before sending. A phone photo sent whole is several megabytes of base64 per turn, it exceeds Claude’s per-image cap, and it is resent with every later request until the turn ages out of the kept history; past that size neither model sees more detail. The transcript shows the thumbnail on the user’s echoed turn, not as an optimistic local copy, for the same reason typed text waits for the echo. The echo carries only an image count and the client matches its own thumbnails to it, so the daemon never sends the image back. Every typed turn gets exactly one answer, its echo or a refusal flagged as such, so a refused turn cannot shift the next turn’s thumbnails onto the wrong echo.

Forwarding images to agents. send_image_to_pane is how a pasted image reaches a pane (“send this to the auth fix”). The daemon numbers each pasted image across the conversation and tells the model the number in the turn text, since the model cannot otherwise name an image from an earlier turn. The numbered copies are kept only for as many turns as the chat model keeps, so the memory the numbering holds is the history’s own bound. The tool goes through the same consent card as type_in_pane, with a thumbnail of the image on it, so nothing reaches the pane before Send, and the approval is bound to the pane’s process like any other. Delivery is not a second implementation: the tool calls the same function as the pane composer’s image endpoint (server.attach_image), so the agent gets the image in the form a manual paste produces (inline via the clipboard, or the staged path when the session is locked), followed by the caption and Enter. Rejected alternative: a loopback HTTP call to the endpoint, which would add an auth hop and lose the pane-identity binding. In a voice session the tool acts without a card, like that session’s other actions. The audit record carries the image type, size and target, never the bytes.

v0.2 candidates (explicitly out of scope now)

Pane creation (“open claude code in a new pane in ~/src/x”) once the create-pane endpoint exists; the approval gate; a wake word; live (in-session) cost display rather than the current fold-in-at-end; carrying usage across a mid-session reconnect (today a new connection restarts the count on every provider).