Run the executor

Build and start:

pnpm --filter @eyeball/executor build
EYEBALL_API_KEYS="eyeball_local_key:proj_local" \
  PORT=3000 \
  pnpm --filter @eyeball/executor start

Use liveness to confirm that the process can answer HTTP requests, and readiness to decide whether it should receive execution traffic:

curl http://127.0.0.1:3000/health
curl http://127.0.0.1:3000/ready

GET /health is dependency-free liveness and remains 200 while the executor process is responsive. GET /ready is the fail-closed traffic-admission check. It returns 200 only when every dependency reports ok, or 503 with a redacted database, migrations, credentials, and queue breakdown. Both routes are public so an orchestrator does not need a project API key; readiness responses use Cache-Control: no-store and never include connection strings, credential material, endpoint URLs, or underlying errors. Each dependency check has an executor-owned 10-second deadline, so a stalled dependency cannot leave the request open indefinitely.

Without EYEBALL_DATABASE_URL, the process-local database and migration checks are ready once the runtime has booted. With Postgres enabled, readiness verifies both a live database query and exact parity between the applied Drizzle journal and the executor's committed migrations. The credential check validates the selected local provider's invariants; Cloud mode sends an authenticated, schema-valid impossible sentinel through the configured resolver and accepts only its redacted missing-credential terminal response. This reaches the resolver's billing and vault dependencies without reading a real credential. The queue check requires the task system to be started, admitting submissions, accepting claims, and able to access the durable job table and its required admission permissions.

EYEBALL_API_KEYS is a comma-separated keyring. Use key:projectId for project-wide authority or key:projectId:userId to pin a key to one end user. A pinned key rejects conflicting request-body, header, query, and MCP metadata identities. Duplicate keys and malformed entries fail startup.

The stock src/server.ts creates the executor runtime with:

• the default catalog and adapters;
• the credential provider selected by EYEBALL_CREDENTIALS (mock by default; env, local-vault, or cloud);
• process-local stores when EYEBALL_DATABASE_URL is absent;
• durable Postgres stores when EYEBALL_DATABASE_URL is present;
• no development connection routes.

To connect real accounts you do not need a custom entry point: set EYEBALL_CREDENTIALS to env or local-vault and supply that mode's variables on this stock server. See Connect real providers. Building your own entry point that passes a devVault to createExecutorApp is only for development connection-injection routes and must never be exposed in a production multi-tenant service.

Hosted mode against eyeball Cloud#

The stock runtime can verify Cloud-issued API keys dynamically, resolve hosted OAuth connections, and enforce Cloud execution usage without a custom server entry point. The three bridges are independent: configure only the ones the deployment needs, or configure all three for a complete hosted data plane. When hosted credentials are selected, /ready fails closed until the resolver URL and internal secret are configured and the exact resolver endpoint returns the expected missing-credential response for the authenticated sentinel probe.

export EYEBALL_API_KEYS="ey_break_glass:proj_operations"
export EYEBALL_KEY_VERIFY_URL="https://cloud.example.com/internal/keys/verify"
export EYEBALL_CREDENTIALS=cloud
export EYEBALL_CREDENTIALS_URL="https://cloud.example.com/internal/credentials/resolve"
export EYEBALL_USAGE_URL="https://cloud.example.com"
export EYEBALL_INTERNAL_API_SECRET='<shared-32+-character-internal-secret>'
pnpm --filter @eyeball/executor start

Cloud endpoint variables must be absolute HTTPS URLs without embedded credentials, queries, or fragments. HTTP is accepted only for explicit loopback development. The executor sends bearer-authenticated, no-store JSON POST requests and never places an API key in a URL. Credential resolution and usage requests time out after 10 seconds by default and expose only generic, redacted failures.

When a remote voice worker serves more than one trusted user, configure EYEBALL_VOICE_SESSION_GRANT_SECRET on the executor and nowhere else. The executor then issues a fresh capability before worker admission and verifies its durable identity and revocation on every child /v1/execute request. See Voice worker for the v2 wire contract and static single-user fallback.

Successful key checks are cached for 60 seconds by default, so revocation reaches an already warm executor within that documented window. Invalid or revoked results are cached for only 5 seconds. Cache entries use the key's SHA-256 digest, not the plaintext key. Verifier transport and protocol failures are never cached: the executor fails closed with a retryable 401, then tries the control plane again on the next request.

Cloud performs OAuth refresh before /internal/credentials/resolve returns. The executor validates the returned ResolvedCredential union and passes the fresh credential through the normal adapter path; it never owns refresh tokens. Keep request-body logging disabled across both services and every proxy between them.

When usage enforcement is configured, the executor reserves one execution after project rate-limit admission and before it allocates an execution record or dispatches an adapter. A quota denial returns HTTP 429 with the closed-taxonomy code rate_limited, a quota message, and no Retry-After header: a plan change can unblock the project immediately, so the executor does not invent a period-end delay. The denied request allocates no execution record and emits no terminal report.

Usage-service transport and protocol failures resolve as follows:

Hosted credential composition defaults to strict enforcement because a degraded usage control plane must not become an unbounded unbilled-execution path. Self-hosted deployments retain the existing availability-first default, and the resolved mode and source are logged at startup; an explicit hosted fail-open relaxation is warning-level. Invalid override values fail startup. An explicit Cloud quota denial is always enforced in every mode.

Every execution that reaches adapter dispatch is placed in a local outbox when it becomes terminal and reported at least once in batches of up to 50; Cloud deduplicates the stable usage identity exactly once. Failed deliveries use exponential backoff and are never silently dropped, including after the alert threshold. With EYEBALL_DATABASE_URL, the usage_outbox table survives executor restarts. Without Postgres, the outbox is process-local, matching the other zero-config defaults. Shutdown performs a bounded best-effort drain using EYEBALL_USAGE_DRAIN_TIMEOUT_MS.

If an admitted execution cannot reach adapter dispatch—for example, credential resolution fails—the executor releases the reservation instead of reporting billable usage. Release is best-effort; Cloud's reservation TTL remains the fallback when the release request cannot be delivered.

Postgres persistence#

Set EYEBALL_DATABASE_URL to persist execution records and idempotency reservations; lease-fenced execution, webhook-selection, and webhook-delivery jobs; immutable webhook payload/endpoint snapshots and delivery attempts; trigger subscriptions, cursors, dedup claims, and redacted trigger-event history; staged-file metadata and bytea content; terminal usage reports awaiting delivery; and the voice-agent aggregate: stable agent heads, append-only immutable revisions, number bindings, executor-side session pointers, message receipts, voice_agent_session_observers, and voice_webhook_sources. The stock runtime uses a pg pool capped at five connections and applies the committed executor Drizzle migrations before accepting traffic.

EYEBALL_DATABASE_URL="postgresql://eyeball:eyeball_dev@127.0.0.1:5432/eyeball" \
EYEBALL_API_KEYS="eyeball_local_key:proj_local" \
  pnpm --filter @eyeball/executor start

You can also apply migrations without starting the executor:

EYEBALL_DATABASE_URL="postgresql://eyeball:eyeball_dev@127.0.0.1:5432/eyeball" \
  pnpm db:migrate

For local development, the root Compose file provides Postgres only; it is optional and is not used by CI or the store contracts:

docker compose up -d postgres

The contract suite runs the same schema, migrations, and Drizzle queries against embedded PGlite, so tests do not require Docker or a live database. Protect the database like any credential store: webhook signing secrets are intentionally available to the delivery worker and therefore reside in the database.

Postgres-backed staged files survive executor restart until expiresAt. Logical expiry is enforced on every metadata/content read and list. At durable startup the runtime drains expired rows in 100-row batches; while healthy, it also runs a non-overlapping once-per-minute sweep that deletes at most 100 rows per tick, and reads lazily remove an expired row addressed by ID. The runtime stops and drains an active sweep before closing the database. Public routes return metadata only, and adapter code consumes bytes exclusively through the execution-bound AdapterContext.files resolver. The stock implementation stores bytes in bytea; FileStore remains the replacement seam for a future metadata-plus-object-store backend.

Redacted trigger-event history has its own seven-day retention window, independent of the 24-hour dedup claim. Reads logically exclude rows exactly at expiresAt. Durable startup drains expired history in 100-row batches before polling begins; a separate non-overlapping once-per-minute sweep continues through consecutive bounded 100-row batches until the expired backlog is empty, and shutdown awaits active cleanup before closing Postgres. The memory runtime applies the same logical retention and online sweep but loses all history on restart. Trigger history stores only arrival IDs, stable Eyeball event/subscription/trigger identity, delivery mode, received/occurred times, closed dedup/admission statuses, endpoint target IDs, and expiry. It never stores provider payloads or provider event IDs, provider cursors, ingest or signing secrets, endpoint URLs, credentials, request/response bodies, or headers.

Postgres voice agents use stable UUID-based IDs and a mutable head that points to append-only immutable revision rows. Number bindings and executor-side session pointers reference the exact selected revision, so later agent updates cannot silently change an existing number or session. Message receipts make chat-turn replay durable. The adjacent observer record is deliberately separate from the mostly immutable pointer: it owns the high-churn cursor, phase, retry state, terminal/transcript markers, and lease. Complete voice webhook envelopes live in the shared source table so any replica can reconstruct ID-only webhook work.

Startup constructs the stores, deliverer, worker client, observer, adapters, and engine; binds handlers; recovers durable execution/webhook jobs; reconciles remote observers; and only then starts task-worker claims. Observer reconciliation backfills missing rows, respects healthy leases and persisted retry deadlines, resumes after the last durably handled worker sequence, and drains sessions that became terminal during downtime. Async execution admission returns only after its job row is durable. Webhook retries retain nextRetryAt and serialize by an opaque project/endpoint group, while different endpoints may progress concurrently.

Execution restart behavior is intentionally conservative. Pending work and running work with no dispatch marker can resume. Once dispatch_started_at is committed, a restart cannot know whether the provider accepted the request, so recovery records a terminal execution_interrupted failure and does not call the adapter again. Terminal usage and webhook effects are reconciled idempotently before the job is acknowledged.

When EYEBALL_DATABASE_URL is absent, execution and webhook jobs use the same serializable state machine in memory but do not survive a process restart. Redacted trigger-event history, staged uploads, voice-agent definitions/revisions, number bindings, session pointers, observer state, voice webhook sources, and message receipts also remain process-local and become unavailable after restart.

Provider base-URL overrides use the manifest-declared environment variable for that adapter. Keep those variables trusted; requests cannot supply an arbitrary origin.

For redacted JSON logs, optional OTLP traces and metrics, and graceful exporter flushing, see Observability.