Queue Forecasting Dashboard

Generated 2026-11-09 02:30:57Z · refreshes every 1h · Project Status · Predictions · Aggregations

Today (UTC)

14,719 pending 15,089 resolved 14,133 completed 798 failed
Hour (UTC)PendingStarted ResolvedCompletedFailed Activity
00:00 5,698 5,685 5,552 5,071 423
01:00 7,768 7,809 7,812 7,484 317
02:00 1,253 1,161 1,725 1,578 58

Data Collection Health

Table Sizes

TableRowsNewestAge
queue_forecast_daily_health 253 2026-11-09 01:42:22Z 49m ago
queue_forecast_run_predictions 12,302,055 2026-11-09 02:28:17Z 3m ago
queue_forecast_task_runs 15,215,940 2026-11-09 02:28:14Z 3m ago
queue_forecast_tasks 14,957,905 2026-11-09 02:28:14Z 3m ago
queue_forecast_worker_counts 7,544,966 2026-11-09 02:25:46Z 5m ago
queue_forecast_worker_pools 96,755 2026-11-08 12:52:42Z 13.6h ago

Stream Freshness

StreamLatestAge
pending_at 2026-11-09 02:30:44Z 12s ago OK
started_at 2026-11-09 02:30:45Z 12s ago OK
resolved_at 2026-11-09 02:30:42Z 15s ago OK
enriched_at 2026-11-09 02:30:45Z 12s ago OK
worker_sample 2026-11-09 02:25:46Z 5m ago OK
daily_health 2026-11-09 01:42:22Z 49m ago OK

Ingestion Windows

WindowPendingStartedResolved
5m 269 224 107
15m 833 818 735
1h 3,187 3,157 4,890
6h 43,021 42,909 43,825
24h 137,506 130,661 137,193

Open Issues

MetricValue
daily_health_rows_last_7d 7
unenriched_tasks 32,414
unresolved_runs 1,492
unresolved_runs_older_than_2h 916
worker_samples_last_30m 6

Recent Resolutions (1h)

ReasonCount (1h)
completed4,630
failed139
deadline-exceeded63
canceled16
worker-shutdown9
intermittent-task1

Training Results

Latest: trainer/data/models/2026-11-03/
trained 5.0d ago · as_of 2026-11-03 · lightgbm
train: 6,456,648 rows (2026-09-28..2026-10-28) · val: 291,962 · holdout: 1,146,807
ModelBaselineDelta
MAE 3.1m 3.3m -5.9%
within-2x 82.6% 80.3% —
p90 coverage 86.3% — —
Per-day holdout breakdown
DaynMAEw/in 2xp90 cov
2026-10-29 291,766 3.4m 80.4% 84.5%
2026-10-30 279,778 3.4m 80.7% 84.9%
2026-10-31 120,847 2.9m 84.7% 84.4%
2026-11-01 145,647 3.7m 82.3% 86.2%
2026-11-02 255,504 2.1m 86.5% 90.8%

Walk-Forward Evaluation

Cohort Baseline MAE Model MAE Δ MAE % Model within-2x Δ within-2x pp p90 cov Hold rows
2026-04-14 2.4m 2.4m 0.1% 90.9% -0.5pp 89.6% 935,671
2026-04-15 2.4m 2.3m -3.4% 91.0% -0.2pp 89.5% 842,633
2026-04-16 2.5m 2.3m -9.1% 90.5% +0.5pp 89.4% 740,907
2026-04-17 2.6m 2.4m -7.3% 89.8% +0.4pp 89.6% 797,924
2026-04-18 2.6m 2.5m -3.7% 89.6% +0.1pp 89.8% 882,507
2026-04-19 2.6m 2.6m -1.2% 88.9% -0.1pp 90.1% 731,367
2026-04-20 2.4m 2.5m 4.9% 89.0% -0.3pp 90.1% 628,458
2026-04-21 2.3m 2.6m 13.0% 88.7% -1.0pp 88.9% 661,844
2026-04-22 2.2m 2.4m 12.3% 89.7% -0.5pp 89.5% 685,783
2026-04-23 2.3m 2.4m 5.6% 89.1% +0.4pp 88.0% 757,919
2026-04-24 2.8m 2.9m 5.0% 88.1% +0.4pp 85.7% 870,087
2026-04-25 2.8m 2.8m 2.8% 88.6% +0.6pp 85.7% 936,164
2026-04-26 2.8m 2.9m 2.7% 88.8% +0.7pp 85.9% 763,702
2026-04-27 3.1m 3.1m 1.1% 87.8% +0.9pp 84.1% 559,122
2026-04-28 2.9m 2.9m -0.3% 89.1% +0.7pp 84.5% 531,856
2026-04-29 2.3m 2.3m 2.4% 90.6% +0.5pp 91.2% 628,767

Daily Health (last 60d)

85%87%90%92%94% 09-1009-2210-0410-1510-2711-08
DateTotalComp%Exc% Wait p99Cap p50Util ExcStuckWaitVolComp Cap↓Cap↑Util↓Samp Anomaly
2026-11-08 125,861 89.5% 0.2% 17.1m 989 89.8% 2 flags
2026-11-07 139,243 89.3% 0.3% 26.2m 1,076 90.0% 2 flags
2026-11-06 318,080 89.3% 1.3% 3.4h 3,011 92.4% OK
2026-11-05 339,974 90.7% 1.2% 3.5h 3,132 91.3% OK
2026-11-04 336,668 90.2% 1.7% 52.2m 2,632 91.3% OK
2026-11-03 329,156 83.7% 1.2% 51.3m 2,876 90.8% OK
2026-11-02 286,393 89.2% 1.9% 1.3h 2,705 91.6% OK
2026-11-01 165,791 87.8% 1.4% 10.7h 1,603 93.0% 1 flag
2026-10-31 150,499 80.3% 0.5% 53.6m 1,210 91.1% 3 flags
2026-10-30 325,623 85.9% 1.4% 1.8h 2,911 91.9% OK
2026-10-29 323,191 90.3% 1.9% 1.6h 2,809 91.4% OK
2026-10-28 320,461 88.7% 1.7% 1.2h 2,514 91.3% OK
2026-10-27 307,899 90.6% 2.0% 2.1h 2,710 90.8% OK
2026-10-26 241,097 93.0% 2.0% 54.0m 1,820 89.0% OK
2026-10-25 51,090 89.0% 0.8% 17.3m 394 88.9% 2 flags
2026-10-24 107,569 87.0% 2.2% 21.3m 860 90.7% 2 flags
2026-10-23 209,939 86.5% 2.3% 41.0m 1,983 90.7% OK
2026-10-22 267,970 87.7% 2.3% 1.0h 1,966 91.7% OK
2026-10-21 298,367 86.1% 1.9% 45.4m 2,554 90.9% OK
2026-10-20 339,726 88.2% 2.5% 4.6h 3,186 91.8% OK
2026-10-19 327,033 93.0% 2.0% 1.6h 3,122 91.4% OK
2026-10-18 107,834 92.1% 0.9% 26.2m 904 89.6% 2 flags
2026-10-17 165,505 86.6% 1.1% 39.6m 1,194 91.7% 1 flag
2026-10-16 360,089 87.4% 1.9% 3.3h 3,300 91.7% OK
2026-10-15 343,168 87.9% 2.0% 2.4h 3,142 91.6% OK
2026-10-14 331,162 89.6% 2.6% 1.7h 3,233 91.1% OK
2026-10-13 281,165 89.5% 2.3% 1.0h 2,648 91.2% OK
2026-10-12 205,942 92.7% 1.9% 1.2h 1,613 90.2% OK
2026-10-11 111,448 88.6% 0.9% 30.6m 711 88.6% 2 flags
2026-10-10 126,441 90.8% 1.4% 40.5m 1,088 90.7% 2 flags
2026-10-09 339,477 87.4% 2.1% 1.4h 3,486 91.8% OK
2026-10-08 336,724 86.2% 2.3% 1.4h 2,708 90.2% OK
2026-10-07 345,146 86.9% 3.5% 2.5h 3,148 91.1% OK
2026-10-06 325,374 90.2% 2.7% 2.3h 3,174 90.1% OK
2026-10-05 244,438 92.4% 2.3% 1.7h 3,839 91.3% OK
2026-10-04 17,294 86.0% 2.4% 26.7m 389 86.5% 3 flags
2026-10-03 100,037 90.8% 1.1% 37.2m 754 90.0% 2 flags
2026-10-02 229,032 86.7% 2.3% 1.0h 2,374 91.5% OK
2026-10-01 226,090 86.2% 2.6% 52.0m 2,236 90.3% OK
2026-09-30 259,716 85.4% 2.2% 55.2m 2,236 90.6% OK
2026-09-29 181,751 86.8% 1.6% 38.6m 1,546 90.0% OK
2026-09-28 201,461 92.2% 1.7% 51.1m 1,846 89.6% OK
2026-09-27 95,753 91.0% 1.0% 43.7m 583 86.1% 2 flags
2026-09-26 120,087 79.5% 1.7% 1.2h 1,184 89.9% 3 flags
2026-09-25 336,832 70.9% 1.9% 1.6h 2,565 90.2% 1 flag
2026-09-24 405,494 78.7% 2.1% 6.5h 3,764 88.8% OK
2026-09-23 345,604 88.9% 2.4% 5.6h 3,037 90.0% OK
2026-09-22 331,673 88.5% 2.5% 6.5h 3,471 90.2% OK
2026-09-21 308,336 91.0% 1.8% 2.4h 3,178 91.1% OK
2026-09-20 101,061 94.3% 0.7% 35.1m 703 90.3% 2 flags
2026-09-19 142,902 75.0% 1.3% 1.6h 920 91.6% 3 flags
2026-09-18 372,857 81.5% 2.2% 3.8h 3,606 91.9% OK
2026-09-17 329,044 83.5% 2.4% 2.9h 3,232 91.6% OK
2026-09-16 374,675 84.7% 1.9% 3.6h 3,218 92.0% OK
2026-09-15 302,666 88.8% 1.9% 6.3h 2,858 91.5% 1 flag
2026-09-14 344,459 88.8% 1.5% 5.7h 3,358 92.0% 1 flag
2026-09-13 124,370 89.9% 0.9% 55.0m 1,078 91.7% 2 flags
2026-09-12 135,385 82.8% 0.8% 29.9m 1,139 91.8% 3 flags
2026-09-11 320,475 85.3% 2.3% 1.7h 2,940 91.7% OK
2026-09-10 352,733 84.8% 2.1% 2.5h 3,535 92.1% OK