Voice worker

apps/voice-worker is the persistent control-plane implementation of eyeball.voice-worker.v2. The TypeScript executor owns agent definitions, canonical schemas, credentials, execution records, and signed webhooks. The Python worker owns durable session state, a gap-free event log, chat turns, and restart-safe child dispatch. Pipecat, Twilio, and LiveKit code paths are present for provider certification, but the repository does not yet claim working live-call capability.

client → executor → versioned worker control plane
                      ├─ Twilio or LiveKit integration path (uncertified)
                      ├─ Deepgram → Anthropic → ElevenLabs path (uncertified)
                      ├─ durable SQLite session + events
                      └─ allowlisted tool → executor /v1/execute

The provider-integration code follows Pipecat's documented pipeline, function-calling, Twilio WebSocket, and transport initialization patterns. Pipecat and every provider SDK load lazily, so the account-free suite does not instantiate or verify those SDK paths.

Configure the trust boundary#

Give the executor one project key for normal clients and configure a distinct signing secret for short-lived voice-session capabilities. The secret remains executor-only; every issued capability is bound to one project, user, executor-owned session ID, expiry, and immutable canonical-tool allowlist. Agent sessions cap maxDurationSeconds at 3,600, so the existing 60-second shutdown buffer bounds every signed grant to 61 minutes.

EYEBALL_API_KEYS="ey_project:proj_local" \
EYEBALL_VOICE_WORKER_URL="https://voice-worker.example.com" \
EYEBALL_VOICE_WORKER_TOKEN="replace-with-at-least-32-random-bytes" \
EYEBALL_VOICE_SESSION_GRANT_SECRET="replace-with-a-distinct-32-byte-random-secret" \
  pnpm --filter @eyeball/executor start

Configure the worker with the same control token. In session-grant mode, EYEBALL_VOICE_WORKER_KEY can remain unset:

cd apps/voice-worker
cp .env.example .env
# Fill .env, then:
docker compose --env-file .env up --build

The service credentials have different directions:

Every /v1/sessions/* control request carries the shared token plus X-Eyeball-Voice-Worker-Version: eyeball.voice-worker.v2. Child calls return through the normal /v1/execute route with a stable X-Eyeball-Execution-Id, exact X-Eyeball-Voice-Session-Id, and voice-session:<session>:event:<sequence> idempotency key. Grant-authenticated calls are accepted only on synchronous /v1/execute requests whose user, session, and canonical tool match the signed capability. Static pinned keys retain their separate compatibility path.

The worker excludes the opaque grant from durable request snapshots, events, transcripts, and health responses. A dedicated SQLite authorization row retains it only while needed for crash recovery and clears the bearer on terminal transition. The executor persists grant identity, expiry, and revocation with the session pointer when EYEBALL_DATABASE_URL is configured. An explicit executor stop revokes immediately; a terminal event does the same before the durable observer cursor advances. If the executor is down when that event arrives, startup reconciliation replays it and revokes the grant idempotently; the terminal worker has already erased its own bearer.

Provider-certification settings#

Provider secrets remain process environment or secret-manager inputs. They are never written into the immutable agent snapshot, SQLite session row, ordered event log, transcript, or executor response. The executor grant is not a provider credential; its dedicated active-session row and terminal erasure are described above.

The agent's llm.model currently passes directly to Anthropic. A hosted model registry and per-project model credential resolution remain separate control-plane work.

Deploy on Fly for certification#

The checked-in fly.toml keeps one 2 GB Machine running, mounts a durable volume, checks /health, and gives SIGTERM 35 seconds to drain.

cd apps/voice-worker
fly volumes create voice_worker_data --region <region> --size 10 --app <app>
fly secrets set --app <app> \
  EYEBALL_VOICE_WORKER_TOKEN='...' \
  EYEBALL_EXECUTOR_URL='https://executor.example.com' \
  EYEBALL_VOICE_PUBLIC_URL='https://<app>.fly.dev' \
  ANTHROPIC_API_KEY='...' \
  DEEPGRAM_API_KEY='...' \
  ELEVENLABS_API_KEY='...'
fly deploy --app <app> --region <region>

Set the Twilio secrets, LiveKit secrets, or both for the transports you use. Pipecat's Fly deployment guide explains the platform-level media considerations; this repository's fly.toml additionally preserves the single-writer SQLite boundary.

Durability and recovery#

Before calling the executor, the worker commits a tool_call event and pending record containing the canonical tool, input, sequence, and stable session-bound execution ID. It commits tool_result before clearing the pending record. On restart, fake sessions resume their cursor and unresolved tool calls reuse the exact executor identity and idempotency key. Recovery computes the remaining maxDurationSeconds window from the original session creation time; a restart never grants a fresh duration budget.

Session events use monotonically increasing, gap-free sequence numbers. voice-agents.get_agent_session polls incrementally, and get_session_transcript normalizes the complete durable history. When Postgres is configured, the executor stores a lease-fenced cursor whose value means “last sequence with all required executor effects durable.” For selected events it persists the complete voice webhook source before deterministic work admission, then checkpoints the cursor. Startup resumes after that cursor; terminal finalization separately rereads the complete worker history from sequence zero so a transcript includes turns from before and after an executor restart without duplicating event admission.

Twenty consecutive retryable observation failures, or one non-retryable protocol failure, produce a durable exhausted observer. The executor emits redacted voice.observer_exhausted telemetry and one deterministic voice.observer.failed webhook. That executor-owned signal is outside the worker's ordered sequence and does not change the authoritative session state.

The configured carrier-recovery policy marks an interrupted Twilio or LiveKit session failed instead of placing a duplicate call or joining twice. Graceful SIGTERM stops admission, waits for active tasks, and marks sessions that outlive the drain deadline abandoned. The carrier behavior still requires end-to-end certification.

Health and provider-free tests#

GET /health is intentionally public for container probes. It verifies SQLite access and reports admission state, active sessions, whether Pipecat imports, and whether the configured model/STT/TTS environment variables are present. media.liveReady is configuration readiness only; it does not contact providers or certify a media path.

Run the account-free contract suite with the lightweight dependencies:

cd apps/voice-worker
python -m pip install -e '.[dev]'
PYTHONPATH=src python -m pytest

The fake transport is accepted only with both EYEBALL_VOICE_MEDIA_MODE=fake and EYEBALL_VOICE_ALLOW_FAKE_TRANSPORT=true. Tests prove ordered HTTP event pages, stable child identity, restart replay, allowlist enforcement, chat concurrency/idempotency, the Anthropic tool-result message shape through an injected fake SDK, control authentication, and deterministic Twilio request/media-URL construction. They do not prove provider acceptance or audio behavior.

For a provider-free control-plane smoke test, start the worker with the test-only transport explicitly enabled and no provider credentials:

EYEBALL_VOICE_MEDIA_MODE=fake \
EYEBALL_VOICE_ALLOW_FAKE_TRANSPORT=true \
EYEBALL_VOICE_WORKER_TOKEN='replace-with-a-long-random-token' \
  docker compose --env-file .env up --build

Then create a fake session over the v2 HTTP contract, wait for its terminal state, and verify the persisted event page has a gap-free sequence. The supplied assistant turn means this path does not call a model provider.

export {};

const workerUrl = process.env.EYEBALL_VOICE_WORKER_URL!.replace(/\/$/u, "");
const controlToken = process.env.EYEBALL_VOICE_WORKER_TOKEN!;
const contractVersion = "eyeball.voice-worker.v2";
const sessionId = `session_${crypto.randomUUID().replaceAll("-", "")}`;
const terminalStates = new Set(["completed", "failed", "abandoned"]);

const object = (value: unknown): Record<string, unknown> => {
  if (typeof value !== "object" || value === null || Array.isArray(value)) {
    throw new Error("Expected an object");
  }
  return value as Record<string, unknown>;
};
const headers = {
  authorization: `Bearer ${controlToken}`,
  "content-type": "application/json",
  "X-Eyeball-Voice-Worker-Version": contractVersion,
};
const workerRequest = async (
  path: string,
  init?: RequestInit,
): Promise<Record<string, unknown>> => {
  const response = await fetch(`${workerUrl}${path}`, {
    ...init,
    headers: { ...headers, ...init?.headers },
  });
  if (!response.ok) {
    throw new Error(`${init?.method ?? "GET"} ${path} returned ${response.status}`);
  }
  return object(await response.json());
};

const created = await workerRequest("/v1/sessions", {
  method: "POST",
  body: JSON.stringify({
    contractVersion,
    sessionId,
    scope: { projectId: "proj_worker_smoke", userId: "user_worker_smoke" },
    agent: {
      id: "va_worker_smoke",
      revision: 1,
      systemPrompt: "Return the scripted assistant response.",
      llm: { provider: "anthropic", model: "fixture/provider-free" },
      voice: {},
      allowedTools: [],
      guardrails: {
        maxDurationSeconds: 30,
        handoffToHuman: { enabled: false },
      },
      webhooks: { endpointIds: [], transcript: true, events: [] },
      recordingPolicy: {
        mode: "disabled",
        consent: "external",
        retentionDays: 0,
        redactDtmf: true,
      },
      bargeIn: { enabled: true },
    },
    transport: {
      kind: "fake",
      turns: [
        {
          caller: "Is the control plane durable?",
          assistant: "Yes. This response came from the fake transport.",
          delayMs: 0,
        },
      ],
    },
  }),
});
if (created.contractVersion !== contractVersion) {
  throw new Error("Worker returned the wrong contract version");
}

let state = String(object(created.session).state);
for (let attempt = 0; attempt < 100 && !terminalStates.has(state); attempt += 1) {
  await new Promise((resolve) => setTimeout(resolve, 25));
  const snapshot = await workerRequest(`/v1/sessions/${sessionId}`);
  state = String(object(snapshot.session).state);
}
if (state !== "completed") throw new Error(`Fake session ended in ${state}`);

const page = await workerRequest(
  `/v1/sessions/${sessionId}/events?afterSequence=0&limit=200`,
);
const eventsValue = page.events;
const events = Array.isArray(eventsValue) ? eventsValue.map(object) : [];
const sequences = events.map((event) => Number(event.sequence));
if (
  sequences.length === 0 ||
  sequences.some((sequence, index) => sequence !== index + 1) ||
  Number(page.nextSequence) !== sequences.at(-1)
) {
  throw new Error(`Event history is not gap-free: ${sequences.join(", ")}`);
}
console.log({ sessionId, state, sequences });

This smoke test certifies only the provider-free v2 control plane. It does not certify the Twilio, LiveKit, Deepgram, ElevenLabs, or model-provider live paths.

Current operational boundaries#

• Run exactly one worker replica per SQLite volume. Multi-replica scheduling needs a distributed repository and session lease.
• Session grants are the multi-user authority path. If the executor grant secret is unset, the static EYEBALL_VOICE_WORKER_KEY fallback still requires one worker/key boundary per trusted pinned user.
• Without EYEBALL_DATABASE_URL, the executor's default stores keep definitions, immutable revisions, bindings, session pointers, observer state, voice webhook sources, and message receipts in process memory. With the variable configured, the stock Postgres stores make those executor-owned records restart-durable. Live worker session state and events remain worker-owned.
• Observer task handles and network reads remain local to one executor process, while Postgres cursor/phase/retry state is lease-fenced and startup-reconciled. Source-first admission lets any executor replica reconstruct a voice event body from voice_webhook_sources; actual HTTP delivery remains at least once and receivers must deduplicate by event ID.
• Provider-backed session requests currently require recordingPolicy.mode=disabled; no compliant audio-recording and retention backend ships here.
• Direct chat is covered by account-free tests. Outbound Twilio and worker-level LiveKit integration paths are assembled but unverified; the catalog still has no public WebRTC session-entry tool, and inbound Twilio number provisioning/webhook verification remains open in RFC 002.
• The media path targets Pipecat 1.5.0 APIs but has not been exercised or certified against real Twilio, LiveKit, Deepgram, ElevenLabs, or Anthropic accounts in this repository.

The flagship tutorial remains the zero-account proof of the canonical agent, allowlist, child execution, event, and transcript contracts.