Data quality

Hourly monitors for event pipelines, data validity and onboarding regressions, with their current status and baseline.

Current status

run 2026-11-04T13:40:49.303Z

Firing

Suppressed

known shift

Building baseline

baselines start 2026-10-27

Not evaluated

Pipeline health

Capture, ingestion and server outcome delivery. While one fires, overlapping product results still alert and name it as context.

Events stopped arriving

Unavailablesev2events_stale

Threshold: Silent beyond 1.0 h after at least 30 events would normally have arrived (statistical)
Owner: data-platform · routes to data-quality · cooldown 6 h
Rollback: Roll back only a deploy that changed capture, the stats outbox or its configuration; product behaviour is unaffected, so never roll back product code for a pipeline alert alone.

Diagnose Runbook

PostHog query freshness failed (http_429).

Ingestion lag (p95)

Unavailablesev3ingestion_lag

Threshold: Above 30 min with at least 50 events (provisional)
Window: 1 h current
Owner: data-platform · routes to data-quality · cooldown 1 d
Rollback: Roll back only a deploy that changed capture, the stats outbox or its configuration; product behaviour is unaffected, so never roll back product code for a pipeline alert alone.

Diagnose Runbook

PostHog query ingestion-lag failed (http_429).

Server outcome delivery delayed

OKsev2outbox_delivery_delayed

Threshold: Above 1.0 h (operational)
Owner: data-platform · routes to data-quality · cooldown 6 h
Rollback: Roll back only a deploy that changed capture, the stats outbox or its configuration; product behaviour is unaffected, so never roll back product code for a pipeline alert alone.

Diagnose Runbook

Server outcome delivery delayed by segment
Segment	Status	Labels
event=trade_filled,source=outbox	OK	—
event=deposit_completed,source=outbox	OK	—
event=deposit_failed,source=outbox	OK	—
event=onboarding_step_completed,source=outbox	OK	—

Server outcomes dead-lettered

OKsev2outbox_dead_letters

Threshold: Any (operational)
Owner: data-platform · routes to data-quality · cooldown 1 d
Rollback: Roll back only a deploy that changed capture, the stats outbox or its configuration; product behaviour is unaffected, so never roll back product code for a pipeline alert alone.

Diagnose Runbook

Server outcomes dead-lettered by segment
Segment	Status	Labels
event=trade_filled,source=outbox	OK	—
event=deposit_completed,source=outbox	OK	—
event=deposit_failed,source=outbox	OK	—
event=onboarding_step_completed,source=outbox	OK	—

Venue fills missing from the outbox

Low volumesev2reconciliation_drift

Threshold: Rise of at least 50.0% relative to baseline and |z| ≥ 3; at least 30 current and 100 baseline (statistical)
Window: 1 d current, 7 d baseline, read 25 h after it closes
Owner: data-platform · routes to data-quality · cooldown 1 d
Rollback: Roll back only a deploy that changed capture, the stats outbox or its configuration; product behaviour is unaffected, so never roll back product code for a pipeline alert alone.

Diagnose Runbook

Venue fills missing from the outbox by segment
Segment	Status	Current vs baseline	Labels
source=venue_fills	Low volume	0.0% of 12 vs 0.0% of 3; needs 30	—

Data validity and coverage

Contract violations, duplicates, identity joins, volume and the observed (consenting) population.

Required properties missing

Unavailablesev3required_property_gaps

Threshold: Rate above 1.0% with at least 100 observations (contract)
Window: 1 d current, 7 d baseline, read 1 h after it closes
Owner: data-platform · routes to data-quality · cooldown 1 d
Rollback: Revert the instrumentation change of the release that introduced it when the fix is not same-day; reports keep counting people, so no product rollback.

Diagnose Runbook

PostHog query required-properties failed (http_429).

Duplicate terminal outcomes

Unavailablesev2duplicate_completions

Threshold: Rate above 0.50% with at least 50 observations (contract)
Window: 1 d current, 7 d baseline, read 1 h after it closes
Owner: data-platform · routes to data-quality · cooldown 1 d
Rollback: Revert the instrumentation change of the release that introduced it when the fix is not same-day; reports keep counting people, so no product rollback.

Diagnose Runbook

PostHog query duplicates failed (http_429).

Impossible step ordering

Unavailablesev3impossible_ordering

Threshold: Rate above 0.50% with at least 100 observations (contract)
Window: 1 d current, 7 d baseline, read 1 h after it closes
Owner: data-platform · routes to data-quality · cooldown 1 d
Rollback: Revert the instrumentation change of the release that introduced it when the fix is not same-day; reports keep counting people, so no product rollback.

Diagnose Runbook

PostHog query ordering failed (http_429).

Identity split, merged or unidentified

Unavailablesev2identity_fragmentation

Threshold: Rate above 1.0% with at least 50 observations (contract)
Window: 1 d current, 7 d baseline, read 1 h after it closes
Owner: auth · routes to data-quality · cooldown 1 d
Rollback: Revert the instrumentation change of the release that introduced it when the fix is not same-day; reports keep counting people, so no product rollback.

Diagnose Runbook

PostHog query identity failed (http_429).

Unexpected event volume

Unavailablesev3event_volume

Threshold: Change of at least 50.0% against the same weekday window and |z| ≥ 4; at least 50 expected and 2 baseline weeks (statistical)
Window: 1 d current, 28 d baseline, read 1 h after it closes
Owner: data-platform · routes to data-quality · cooldown 1 d
Rollback: Revert the instrumentation change of the release that introduced it when the fix is not same-day; reports keep counting people, so no product rollback.

Diagnose Runbook

PostHog query volume failed (http_429).

Event missing from a release

Unavailablesev2release_coverage

Threshold: Drop of at least 80.0% relative to baseline and |z| ≥ 3; at least 50 current and 200 baseline; baseline rate ≥ 10.0% (statistical)
Window: 1 d current, 7 d baseline, read 1 h after it closes
Owner: data-platform · routes to data-quality · cooldown 1 d
Rollback: Revert the instrumentation change of the release that introduced it when the fix is not same-day; reports keep counting people, so no product rollback.

Diagnose Runbook

PostHog query release-coverage failed (http_429).

Observed population shifted

Unavailablesev3coverage_shift

Threshold: Change of at least 30.0% against the same weekday window and |z| ≥ 4; at least 50 expected and 2 baseline weeks (statistical)
Window: 1 d current, 28 d baseline, read 1 h after it closes
Owner: product-ops · routes to data-quality · cooldown 1 d
Rollback: Not a rollback signal: annotate the consent or release change as a known shift.

Diagnose Runbook

PostHog query coverage failed (http_429).

Server outcome coverage shifted

Low volumesev3server_coverage_shift

Threshold: Change of at least 30.0% relative to baseline and |z| ≥ 3; at least 30 current and 100 baseline (statistical)
Window: 1 d current, 7 d baseline, read 25 h after it closes
Owner: product-ops · routes to data-quality · cooldown 1 d
Rollback: Not a rollback signal: consent grants and withdrawals move it.

Diagnose Runbook

Server outcome coverage shifted by segment
Segment	Status	Current vs baseline	Labels
source=venue_fills	Low volume	70.6% of 17 vs 73.6% of 106; needs 30	—

Onboarding regressions

Mature cohorts only, compared with their own trailing baseline. Low-volume, immature and shifted segments are labeled instead of alerted.

Onboarding conversion regression

Unavailablesev2onboarding_conversion

Threshold: Drop of at least 20.0% relative to baseline and |z| ≥ 3; at least 100 current and 300 baseline (statistical)
Window: 7 d current, 21 d baseline, read 25 h after it closes
Owner: product-ops · routes to product-ops · cooldown 1 d
Rollback: Roll back the product release named in the segment only after the pipeline and coverage monitors are clear and the drop reproduces on the dashboard; otherwise investigate first.

Diagnose Runbook

PostHog query conversion failed (http_429).

Sign-in failures rising

Unavailablesev2auth_failures

Threshold: Rise of at least 50.0% relative to baseline and |z| ≥ 3; at least 50 current and 200 baseline (statistical)
Window: 1 d current, 7 d baseline, read 1 h after it closes
Owner: auth · routes to product-ops · cooldown 12 h
Rollback: Roll back the product release named in the segment only after the pipeline and coverage monitors are clear and the drop reproduces on the dashboard; otherwise investigate first.

Diagnose Runbook

PostHog query auth-failures failed (http_429).

Funding provider failures rising

Unavailablesev2funding_provider_failures

Threshold: Rise of at least 50.0% relative to baseline and |z| ≥ 3; at least 30 current and 100 baseline (statistical)
Window: 1 d current, 7 d baseline, read 1 h after it closes
Owner: funding · routes to product-ops · cooldown 12 h
Rollback: Roll back the product release named in the segment only after the pipeline and coverage monitors are clear and the drop reproduces on the dashboard; otherwise investigate first.

Diagnose Runbook

PostHog query provider-failures failed (http_429).

Deposits stuck without an outcome

Low volumesev2funding_stuck

Threshold: Rise of at least 50.0% relative to baseline and |z| ≥ 3; at least 30 current and 100 baseline (statistical)
Window: 1 d current, 7 d baseline, read 25 h after it closes
Owner: funding · routes to product-ops · cooldown 12 h
Rollback: Roll back the product release named in the segment only after the pipeline and coverage monitors are clear and the drop reproduces on the dashboard; otherwise investigate first.

Diagnose Runbook

PostHog query funding-stuck failed (http_429).

Deposits stuck without an outcome by segment
Segment	Status	Current vs baseline	Labels
provider=daimo,source=server	Low volume	100.0% of 3 vs 100.0% of 2; needs 30	—

Time to first live fill (p90) rising

Unavailablesev3time_to_activation

Threshold: p90 at least 1.5× baseline with at least 30 current and 100 baseline conversions (provisional)
Window: 7 d current, 21 d baseline, read 169 h after it closes
Owner: product-ops · routes to product-ops · cooldown 1 d
Rollback: Roll back the product release named in the segment only after the pipeline and coverage monitors are clear and the drop reproduces on the dashboard; otherwise investigate first.

Diagnose Runbook

PostHog query activation-time failed (http_429).

Known shifts

Consent, coverage and instrumentation changes. Comparisons across them are labeled and their alerts suppressed.

No known shifts annotated.

Reading these monitors

Windows are read 1 h after they close to allow for expected ingestion lag; baselines start at the first production instrumentation (2026-10-27).
Statistical thresholds compare each segment with its own trailing baseline; they are not conversion targets.
All counts are consenting, observed people or events. A coverage shift is not a change in total traffic.
Venue reliability (trade rejections, Papertrade relayer and RPC errors) is monitored separately under #4565.
Funnel definitions are in the onboarding reports.