Technical architecture · production design

How Prod AI runs in production.

Five planes with one job each. The control plane decides, sandboxes run untrusted code, the runtime serves live apps, and every model call and every agent action passes a gateway that can price it, cap it or ask a person first. The prototype you can click today implements the product surface of this design; the table at the end says exactly which parts are real.

Shown at native size (1,824 px wide). On a narrower screen, scroll sideways.

Open full size
Prod AI production architecture: people, edge, control plane, sandbox plane, runtime plane and data platform, with every service connection, eight numbered flows labelled with their transports, an outline marking one cell (the regional and global pieces sit outside it), and a legend.Each numbered badge can be focused. Focusing or hovering a badge highlights the connections of that flow and shows its description in the panel at the top left.

Trace a flow

Tap, hover or focus a numbered badge (Tab works) to light up its path through the system. The step appears here.

Outside servicesPeopleEdgeregional · ingressControl planestateless · KubernetesSandbox planeuntrusted codeRuntime planelive appsRegionalshared by its cellsData + platformper cellLegendhow to read this diagram
Identity providers

Google, GitHub, SAML SSO

Stripe

Plans and invoices

Model providers

Anthropic · OpenAI · Google · open models (vLLM, Bedrock)

GitHub

Repos, pull requests, checks, webhooks

Vercel

Deploy target

Slack + email

Approval messages

Builder studio

One project for the people who describe apps and the people who code them

PlainSettingsCode
Teammates

Handoffs arrive with the object, history and diff

Your editor

Cursor, Claude Code or VS Code on the same repo

People using live apps

Public URLs and custom domains

CDN + WAF

Cloudflare: static assets, WAF, bot and DDoS protection

API gateway

Regional Envoy: JWT, rate limits, quotas, cell routing

Preview proxy

{project}.preview → sandbox port. WebSockets and HMR, signed cookie, wakes sleeping sandboxes

Realtime hub

SSE and WebSocket fan-out over NATS: build steps, logs, presence

App router

Custom domains, automatic TLS, live traffic to the runtime

Budget + billing

Quote before work, meter after, caps, refunds

Web app + BFF

Next.js in each cell's cluster: server actions and sessions

Orchestrator

Temporal workflows for plan, build, repair, deploy and import. Durable, resumable, retried

Project service

Blueprints, save points, Work Orders, diffs

Policy + approvals

Tool permissions, House Rules, approvals, delegation grants, audit log

Model gateway

The studio's gateway: routing by task, eval-gated switches, cache-affine failover, BYOK, cost per call

Agent harness

Planner, coder, verifier and repairer share one tool loop with step budgets and doom-loop detection

planactverifyrepair
GitHub service

GitHub App: a branch per Work Order, PRs, two-way sync

Deploy service

Immutable releases, preflight gates, instant rollback

Import + analysis

Clone, detect stack and agents, coverage map, House Rules

Sandbox manager

Schedules microVMs: warm pool, lazy snapshot restore, idle suspend. Eligible work can burst to E2B

Project sandbox

Firecracker microVM per project · 2 vCPU · 4 GB · persistent disk

  • Dev server · Next.js or Vite :3000
  • Agent runtime · Python + Node
  • Tests + rehearsal runner
  • Language servers + repo map
  • File watcher → events
Egress proxy

Allow-listed network. Secrets are added on the way out and never live in the VM

Package mirror

npm and PyPI pull-through cache, scanned. Regional

Prod Cloud

Live apps on Knative, each pod in its own microVM. Scale to zero, no secrets, no direct egress

Agent gateway

The only way out of a live app. Holds the credentials, checks permissions and approval gates, caps budgets, traces every call. Its own model gateway, apart from the studio's

ReadChangeAsk first
Notify

Approval and handoff pings, signed replies

Queues + schedules

Triggers, retries, long-running agent tasks

Evals in production

Rehearsals replayed on real traces, drift alerts

Your VPC or on-prem

Same runtime via Helm or Terraform, outbound-only tunnel

Identity

Supabase Auth, memberships, SSO and the workspace → cell directory

Usage warehouse

Tokens, credits and trace summaries (ClickHouse)

Observability

OpenTelemetry traces, logs, LLM spans, SLOs

Postgres

Supabase, one per cell. Projects, Blueprints, ledger, grants, pgvector. RLS on every table

Redis

Preview and app routes, locks, gateway pins. Rate limits are regional

NATS JetStream

Build events, usage, approvals. Replay from any sequence number

Temporal Cloud

Workflow history, timers, task queues. Studio and runtime namespaces

Object storage

Sandbox snapshots, build artefacts, OCI images, repo archives

Secrets vault

KMS-encrypted, regional keys. Only the egress proxy and the gateways decrypt

App databases

Neon Postgres for live apps: a branch per app and per test version

git push / pull · same repoall outbound callsRedis routes · NATS fan-outPostgres RLS · Temporal · S3 · vault · pgvectorsnapshots · egress secrets · eventsNeon · vault · NATSHTTPS + SSEgRPC + TemporalvsockHTTPS + WebSocket (HMR)vsock → NATS → WebSocketgit + REST · webhooks backKnative · Vercel API · mTLS tunnelHTTPS + WebSocket

Line style

  • Solid: request and response over HTTPS, gRPC or vsock
  • Dashed: asynchronous events, webhooks, approval messages
  • Moving dashes, while you trace a flow: the hot path of a request
  • gRPCMono label: the transport, or what a trunk carries

Colour: the plane a line belongs to

  • People: browsers, editors and git
  • Edge: traffic through a gateway or proxy
  • Control plane: service calls, model calls, GitHub
  • Sandbox: microVM lifecycle, tool calls, events, installs
  • Runtime: releases going in, live agents' calls going out
  • Data: the shared bus; each trunk takes its plane's colour
  • Outside: identity federation and billing
  • Dashed box: a plane, its own trust and scaling boundary
  • Long-dash outline: one cell, the blast-radius unit

Numbered badges

1

The eight steps of one request, prompt to production (section 3 of ARCHITECTURE.md). The mono label beside each badge names its transport. Hover or focus a badge to light up its path; the step appears at the top left.

12345678
Fig. 1Five planes, one job each: control, sandboxes, runtime, data and the gateways every model call and agent action passes. Numbers mark the eight flows below.

The eight flows on the diagram

  1. 1Describe

    The studio sends the brief through the API gateway. Nothing runs yet: the planner returns a Blueprint and a priced Work Order.

  2. 2Plan

    The orchestrator starts a durable workflow. The harness asks the model gateway, which picks a model per task and meters every call.

  3. 3Build in a sandbox

    Approved work runs inside the project's microVM: write files, install, run, test, rehearse agents. Every step is checkpointed.

  4. 4Live preview

    The preview proxy maps the project's subdomain to the sandbox dev server, including WebSockets for hot reload.

  5. 5Show the work

    File, test and step events stream back through the realtime hub, so people see progress in plain English, not a spinner.

  6. 6Branch + PR

    Each Work Order becomes a branch and a pull request through the GitHub App. Engineers' pushes sync back into the Blueprint or become code they own.

  7. 7Ship

    Preflight passes, the deploy service builds one immutable release and rolls it out to Prod Cloud, Vercel or your VPC.

  8. 8Governed agents

    The agent gateway is every live app's only way out: credentials, approval gates, caps and traces. Model calls use the runtime's own model gateway.

The agent harness

Plan, act, verify, repair. With a budget and a way out.

The model decides; deterministic code does the rest. The planner writes a Blueprint (structured JSON), code generation turns it into files, and a tool loop edits, runs and tests inside the sandbox. A verifier gates every step. When a check fails, the repairer tries a fix, but only within a step and cost budget. The same error twice means a doom loop: it stops, rolls back to the last save point and hands the problem to a person with the full context. Our own fixes are never billed.

  1. Plan

    Blueprint + Work Order, priced before anything runs

  2. Act

    Tools in the sandbox: edit, run, install, test, browse

  3. Verify

    Types, lint, tests, agent rehearsals, preview smoke

  4. Save point

    Checkpoint, branch commit, streamed to the studio

Repair

Verifier fails: propose a fix with its blast radius. Free, labelled “Our fix”.

Doom-loop stop

Same error signature twice, or 3 attempts: stop, roll back, ask a person.

Stop any time

A person can stop a run; unused credits are refunded automatically.

Sandboxing

One microVM per project, because the code is untrusted.

  • Firecracker microVMs

    Chosen
    Isolation
    Own kernel per project
    Start
    ~150 ms from snapshot
    Fit for agentic apps
    Runs Node and Python agents, real ports, persistent disk
  • gVisor containers

    Fallback
    Isolation
    User-space kernel
    Start
    ~1 s
    Fit for agentic apps
    Good isolation, some syscall and I/O overhead
  • Plain Docker

    Rejected
    Isolation
    Shared host kernel
    Start
    ~1 s
    Fit for agentic apps
    One escape reaches every tenant
  • Browser WebContainers

    Rejected
    Isolation
    Browser tab
    Start
    Instant
    Fit for agentic apps
    Node only: no Python agents, no long-running jobs

Scale

Thousands of people at once, without thousands of idle machines.

5,000

people in the studio at once

~1,500

sandboxes awake (30% active, the rest snapshotted)

~150 ms

resume from a warm snapshot

≤ 3

repair attempts before a person is asked

  • The control plane is stateless; long work lives in durable workflows, so a pod restart never loses a build.
  • Sandboxes suspend to a snapshot after 10 idle minutes and resume in about 150 ms from a warm pool.
  • The model gateway queues by priority per provider and key, fails over between providers, and caches prompts.
  • Events fan out over NATS; Postgres uses row-level security, read replicas and partitioned event tables.

Honest about the seams

What the prototype runs today, and what production adds.

Real runs in the live prototype. Stand-in is simulated or sandboxed, and labelled in the product. Designed is specified in ARCHITECTURE.md and not built yet.

  • Studio, auth, data

    Real

    In the live prototype

    Next.js 16 on Vercel Hobby, Supabase Auth (Google, email link, guest sessions you can keep), Postgres with row-level security on every table. One server-only admin connection writes only the rows people must not: the spend meter, rate limits, budget holds, published sites, abuse reports and published apps' records

    In production

    The studio backend runs in each regional cell (service identities, residency, long-lived streams); Vercel keeps the marketing site. Plus SAML SSO, SCIM and regional data residency

  • Planner

    Real

    In the live prototype

    Claude picks the kind of app. For a business app it plans a structured Blueprint, streamed live, and code expands it deterministically and validates every reference; for anything else it writes real React files, streamed file by file. Guests start from a starter plan

    In production

    Same contract, routed through the model gateway

  • Change requests

    Real

    In the live prototype

    Claude returns typed edits; code resolves names and emits validated operations, with one self-repair retry; each change has a fixed price, Apply and Undo, and makes a new version

    In production

    Same, executed by the harness in a sandbox; parallel Work Orders rebase onto a versioned Blueprint

  • Model gateway

    Model switch and budgets: realRouting gateway: designed

    In the live prototype

    One getModel() seam that picks the model from the environment. Every call passes a rate limit and holds its cost against per-person and site-wide 24-hour budgets; failed calls are metered too. One provider (Anthropic) and no gateway service

    In production

    A gateway service: routing by task, eval-gated switches, failover in the middle of a tool loop, prompt caching, BYOK, and a separate deployment for live apps

  • Approvals and the agent gateway

    Approval gates: realProduction gateway: designed

    In the live prototype

    Claude + AI SDK tool approval, signed by the server: tools marked “Ask first” pause for a person. Try it runs on sample data; helpers in published apps work on that app's records and send email only after a person says yes. Other outside connections aren't real yet, and nothing holds credentials or controls egress

    In production

    Every production tool call passes an egress gateway that holds the credentials, enforces approvals and caps, tracks untrusted content per run, and traces the call

  • Build + repair

    Builds, test runs, repairs: realTool loop in microVMs: designed

    In the live prototype

    A business app's build checks every reference in the plan, compiles the generated code with esbuild, and has Claude play and judge each AI helper's test runs (price shown first); a real failure gets a free fix and a free replay. A code app is compiled and bundled with esbuild in about 15 ms, and Claude repairs real build and runtime errors for free. Generated business-app code is compiled, not executed

    In production

    Full tool loop inside microVMs with verifier and budgets

  • Sandbox + preview

    Code apps' browser sandbox: realServer microVMs: designed

    In the live prototype

    A code app runs in its own page with its own Content Security Policy and an opaque origin (no cookies, no access to the studio), talking to Prod AI only by postMessage. A business app's preview renders the Blueprint with the spec renderer; no generated business-app or imported code runs

    In production

    Firecracker microVM per project behind the preview proxy; imported repos run as themselves, with a preview per exposed port

  • GitHub

    Public repo analysis: realPushes, PRs, sync: sandboxed

    In the live prototype

    Reads any public repo, detects stack and agent frameworks, maps its agents, writes House Rules; pushes and PRs are sandboxed

    In production

    GitHub App with a branch and worktree per Work Order, a merge queue and two-way sync

  • Deploy and published apps

    Public /live URL, records, rollback: realImmutable releases: designed

    In the live prototype

    A public /live/… URL serves a published Blueprint snapshot, or a code app's published build in its sandbox, with rollback, real records (up to 2,000 per app, with undo) and team screens for invited people. Publishing checks the live link answers. Vercel and VPC targets are sandboxed

    In production

    Immutable releases, canary rollout, instant rollback, and a database per app

Service-by-service reasoning, the sequence diagrams and the prototype-to-production map are in ARCHITECTURE.md.