14. Webhooks, notifications and live events

Four ways Nozzle tells you something happened, all fed from one internal publish point. Every event reaches every audience that asked for it:

Cost alerts (§14.5) are a producer: they speak through all four.

14.1 Webhooks

curl -X POST "$BASE/v1/projects/$PROJECT_ID/webhooks" \
  -H "Authorization: Bearer $SESSION_JWT" -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{"url": "https://example.com/hook",
       "description": "prod events",
       "event_types": ["instance.ready", "cost.threshold_reached", "webhook.delivery_failed"]}'

The field is event_types (a body with events is a typed 400). The URL must be https and a public host. The response carries signing_secret (whsec_…) — shown here and on rotate-secret, never again. Webhook management needs a dashboard session; keys are refused with 403.

GET /v1/webhooks/event-types lists everything subscribable (52 types: identity, membership and invitations, keys, instance lifecycle, models and model health, cost thresholds, webhook failures, and webhook.test).

Every delivery is signed. Headers:

v1 is HMAC-SHA256(signing_secret, "<t>.<raw body>"), hex. Verify against the raw body bytes, compare in constant time, and reject a t more than five minutes from now:

import hmac, hashlib, time
def verify(secret: str, raw_body: bytes, header: str, tolerance=300) -> bool:
    parts = dict(p.split("=", 1) for p in header.split(","))
    if abs(time.time() - int(parts["t"])) > tolerance:
        return False
    mac = hmac.new(secret.encode(), parts["t"].encode() + b"." + raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(mac, parts["v1"])

Retries. Any non-2xx or timeout (15 s) is retried with exponential backoff: 2, 4, 8, 16, 32 s … capped at an hour, 6 attempts in all. The sixth failure dead-letters the delivery (state: "dead_lettered") and publishes webhook.delivery_failed — to your other subscriptions, the inbox and the live stream. A failed webhook.delivery_failed delivery never announces itself, so an endpoint subscribed to its own failures cannot loop.

Test and replay.

# a signed webhook.test, now, through the real dispatcher (202 + the delivery)
curl -X POST "$BASE/v1/webhooks/$WEBHOOK_ID/test" \
  -H "Authorization: Bearer $SESSION_JWT" -H "Idempotency-Key: $(uuidgen)"

# attempts, newest first
curl "$BASE/v1/webhooks/$WEBHOOK_ID/deliveries" -H "Authorization: Bearer $SESSION_JWT"

# re-send a past delivery as a NEW delivery (fresh attempt chain, 202)
curl -X POST "$BASE/v1/webhooks/deliveries/$DELIVERY_ID/replay" \
  -H "Authorization: Bearer $SESSION_JWT" -H "Idempotency-Key: $(uuidgen)"

A test ignores the subscription's event_types filter — it tests the endpoint, not the filter. Payload:

{"type": "webhook.test", "event_id": "01a0ca59-…", "subscription_id": "01a0ca59-…",
 "sent_at": "2026-09-22T18:20:32Z",
 "message": "A test delivery from Nozzle. Verify the Nozzle-Signature header against your signing secret."}

Proven 2026-09-22 against a real receiver: 13 deliveries, every signature verified independently; a flaky endpoint (500, 500, 200) succeeded on attempt 3; a dead endpoint (503 ×6) dead-lettered after gaps of 4 / 6 / 10 / 18 / 34 s and fired webhook.delivery_failed; a replay went out as a new delivery id and landed 200.

Tenant-wide facts (a workspace cost alert, a shared model's health) are delivered to matching subscriptions in every project of the tenant.

14.2 Notification channels

Route an event type to Slack, email, APNs, web push or a plain webhook:

curl -X POST "$BASE/v1/tenants/$TENANT_ID/notifications/subscriptions" \
  -H "Authorization: Bearer $SESSION_JWT" -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{"channel": "slack", "event_type": "cost.threshold_reached",
       "target": {"webhook_url": "https://hooks.slack.com/services/…"}}'

target fields per channel: slack webhook_url · email address · apns device_token · web_push endpoint, p256dh, auth · customer_webhook url. event_type may end in .*. Every published event reaches channels — including lifecycle events the gateway publishes in the background (instance.failed, virtual_key.expiring_soon), which before 2026-09-22 reached webhooks only.

14.3 The inbox

What the console bell reads. Tenant-wide — every member sees every row — with read state per reader (a signed-in user, or an API key reading as itself).

A project-scoped key sees its project's rows plus tenant-wide ones.

{
  "items": [{
    "id": "01a0ca5a-3963-7803-8014-a2117db6845f",
    "tenant_id": "01a0ca58-…", "project_id": "01a0ca58-…",
    "kind": "webhook.delivery_failed",
    "severity": "error",
    "title": "Webhook delivery failed",
    "body": "A webhook.test delivery to https://example.com/hook failed after 6 attempts: endpoint returned 503 Service Unavailable",
    "link": "/app/webhooks/01a0ca59-…",
    "resource_type": "webhook_delivery",
    "resource_id": "adf8133f-…",
    "data": { "…the source event's payload…": "" },
    "created_at": "2026-09-22T18:21:45.700732Z",
    "read": false
  }],
  "next_cursor": null, "has_more": false, "unread_count": 4
}

severity is info | notice | warn | error | critical. What rings the bell is a closed list — everything else stays in the audit log:

14.4 The live stream

curl -N "$BASE/v1/events/stream" -H "Authorization: Bearer $KEY_OR_SESSION"

text/event-stream, scope usage:read. One connection per tab is plenty; at most 16 per tenant (the 17th is 429 rate_limit.exceeded). A key streams its own tenant. A session streams the workspace named by X-Nozzle-Tenant, else its earliest one. A browser EventSource can carry the nozzle_access cookie (withCredentials: true) but cannot set headers, so a console that switches workspaces reads the stream with fetch and a streaming body reader instead, sending X-Nozzle-Tenant and Last-Event-ID itself. A missing scope is 403 auth.missing_scope.

Each frame:

id: 1790101273691-1~0-0
event: cost.threshold_reached
data: {"id":"1790101273691-1~0-0","type":"cost.threshold_reached","tenant_id":"…","project_id":null,"occurred_at":"2026-09-22T18:21:13.691082194Z","data":{…}}

plus :ping comments when idle (every ≤15 s). The frame object:

Kinds — each is delivered only if your credential holds its scope, so a narrow key gets a narrow feed rather than an error:

"Envelope" is {"event": "<source event type>", "event_id": "<uuid or null>", "payload": {…}}. A real cost.recorded data:

{"kind": "inference.chat", "cost_micro_cents": 800, "model": "gpt-4.1-mini",
 "request_id": "01a0ca5d-69b6-7640-aff9-e61de86478d4",
 "virtual_key_id": "01a0ca5d-3599-7ff3-a6b9-9abc1c76148d",
 "prompt_tokens": 8, "completion_tokens": 3, "cached_tokens": 0, "estimated": false}

Resume. Reconnect with the last id you saw as Last-Event-ID (an EventSource does this for you) or as ?since=. Frames published while you were away are replayed in order — the buffer holds the last ~1000 frames per tenant. If you were gone longer than that, the stream opens with a stream.resync frame: refetch what you are showing, then carry on live. A malformed token is treated as "from now", never an error.

A frame is a repaint hint, not the record: every fact in it is already in the ledger, audit log or inbox, so a dropped frame costs a late repaint, never lost data.

Shared model lanes: a health change on a platform model goes to every tenant's stream (a catalog repaint) and, as a notification and webhook, to each tenant that called that model in the last 7 days.

14.5 Cost alerts

"Tell me when this workspace — or this one key — has spent $X today / this month." An alert speaks; it does not block. (Hard ceilings are per-key budgets, §11, and plan entitlements, §12.)

curl -X POST "$BASE/v1/cost-alerts" \
  -H "Authorization: Bearer $SESSION_JWT" -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{"name": "daily ceiling", "threshold_micro_cents": 500000000, "window": "day"}'

threshold_micro_cents (1¢ = 1,000,000 µ¢; $5 = 500,000,000). window is day or month, calendar UTC. Add virtual_key_id to watch one key. Scopes billing:read to list, billing:write to change; at most 100 per workspace.

Rules are evaluated every minute. When the window's spend reaches the threshold, the rule fires once for that window and publishes cost.threshold_reached:

{"rule_id": "01a0ca59-…", "rule_name": "probe daily", "virtual_key_id": null,
 "window": "day", "period": "2026-09-22",
 "threshold_micro_cents": 1, "spend_micro_cents": 720}

A key rule's event belongs to the key's project; a workspace rule's belongs to every project.