Queue Forecasting Dashboard

Generated 2026-09-17 06:49:45Z · refreshes every 1h · Project Status · Predictions · Aggregations

Today (UTC)

48,651 pending 53,092 resolved 48,220 completed 2,747 failed
Hour (UTC)PendingStarted ResolvedCompletedFailed Activity
00:00 13,819 13,511 14,469 13,070 592
01:00 6,335 8,226 9,857 8,942 545
02:00 4,003 4,739 4,981 4,342 504
03:00 6,501 6,781 6,662 6,210 335
04:00 6,106 6,263 5,702 5,358 260
05:00 5,408 5,249 6,540 5,783 305
06:00 6,479 5,393 4,881 4,515 206

Data Collection Health

Table Sizes

TableRowsNewestAge
queue_forecast_daily_health 200 2026-09-17 05:47:12Z 1.0h ago
queue_forecast_run_predictions 13,291,386 2026-09-17 06:46:02Z 4m ago
queue_forecast_task_runs 14,576,236 2026-09-17 06:46:03Z 4m ago
queue_forecast_tasks 14,398,868 2026-09-17 06:46:02Z 4m ago
queue_forecast_worker_counts 7,391,526 2026-09-17 06:42:30Z 7m ago
queue_forecast_worker_pools 8,982 2026-09-16 13:19:44Z 17.5h ago

Stream Freshness

StreamLatestAge
pending_at 2026-09-17 06:49:28Z 17s ago OK
started_at 2026-09-17 06:49:30Z 15s ago OK
resolved_at 2026-09-17 06:49:28Z 17s ago OK
enriched_at 2026-09-17 06:49:29Z 15s ago OK
worker_sample 2026-09-17 06:47:30Z 2m ago OK
daily_health 2026-09-17 05:47:12Z 1.0h ago OK

Ingestion Windows

WindowPendingStartedResolved
5m 1,213 711 599
15m 2,321 1,747 1,569
1h 7,278 5,956 5,709
6h 39,215 39,742 40,450
24h 362,328 330,759 362,491

Open Issues

MetricValue
daily_health_rows_last_7d 7
unenriched_tasks 10
unresolved_runs 7,193
unresolved_runs_older_than_2h 4,549
worker_samples_last_30m 6

Recent Resolutions (1h)

ReasonCount (1h)
completed5,012
deadline-exceeded343
failed245
worker-shutdown57
canceled51
claim-expired2
intermittent-task2
internal-error1

Training Results

Latest: trainer/data/models/2026-09-15/
trained 1.2d ago · as_of 2026-09-15 · lightgbm
train: 6,808,747 rows (2026-08-10..2026-09-09) · val: 307,464 · holdout: 1,164,523
ModelBaselineDelta
MAE 1.8m 2.0m -7.4%
within-2x 91.1% 90.1%
p90 coverage 90.3%
Per-day holdout breakdown
DaynMAEw/in 2xp90 cov
2026-09-10 299,040 1.9m 89.8% 89.5%
2026-09-11 273,492 1.9m 89.9% 90.6%
2026-09-12 112,148 1.6m 93.2% 92.1%
2026-09-13 111,834 1.8m 94.5% 88.2%
2026-09-14 305,757 1.8m 91.4% 91.0%

Walk-Forward Evaluation

Cohort Baseline MAE Model MAE Δ MAE % Model within-2x Δ within-2x pp p90 cov Hold rows
2026-04-14 2.4m 2.4m 0.1% 90.9% -0.5pp 89.6% 935,671
2026-04-15 2.4m 2.3m -3.4% 91.0% -0.2pp 89.5% 842,633
2026-04-16 2.5m 2.3m -9.1% 90.5% +0.5pp 89.4% 740,907
2026-04-17 2.6m 2.4m -7.3% 89.8% +0.4pp 89.6% 797,924
2026-04-18 2.6m 2.5m -3.7% 89.6% +0.1pp 89.8% 882,507
2026-04-19 2.6m 2.6m -1.2% 88.9% -0.1pp 90.1% 731,367
2026-04-20 2.4m 2.5m 4.9% 89.0% -0.3pp 90.1% 628,458
2026-04-21 2.3m 2.6m 13.0% 88.7% -1.0pp 88.9% 661,844
2026-04-22 2.2m 2.4m 12.3% 89.7% -0.5pp 89.5% 685,783
2026-04-23 2.3m 2.4m 5.6% 89.1% +0.4pp 88.0% 757,919
2026-04-24 2.8m 2.9m 5.0% 88.1% +0.4pp 85.7% 870,087
2026-04-25 2.8m 2.8m 2.8% 88.6% +0.6pp 85.7% 936,164
2026-04-26 2.8m 2.9m 2.7% 88.8% +0.7pp 85.9% 763,702
2026-04-27 3.1m 3.1m 1.1% 87.8% +0.9pp 84.1% 559,122
2026-04-28 2.9m 2.9m -0.3% 89.1% +0.7pp 84.5% 531,856
2026-04-29 2.3m 2.3m 2.4% 90.6% +0.5pp 91.2% 628,767

Daily Health (last 60d)

86%88%90%92%94% 07-1907-3108-1208-2309-0409-16
DateTotalComp%Exc% Wait p99Cap p50Util ExcStuckWaitVolComp Cap↓Cap↑Util↓Samp Anomaly
2026-09-16 374,675 84.7% 1.9% 3.5h 3,218 92.0% OK
2026-09-15 302,666 88.8% 1.9% 6.3h 2,858 91.5% 1 flag
2026-09-14 344,459 88.8% 1.5% 5.7h 3,358 92.0% 1 flag
2026-09-13 124,370 89.9% 0.9% 55.0m 1,078 91.7% 2 flags
2026-09-12 135,385 82.8% 0.8% 29.9m 1,139 91.8% 3 flags
2026-09-11 320,475 85.3% 2.3% 1.7h 2,940 91.7% OK
2026-09-10 352,733 84.8% 2.1% 2.5h 3,535 92.1% OK
2026-09-09 349,718 85.7% 2.1% 1.6h 3,108 92.3% OK
2026-09-08 295,863 89.1% 1.7% 2.1h 2,698 92.4% OK
2026-09-07 222,808 92.4% 1.1% 57.1m 2,221 92.9% OK
2026-09-06 105,588 82.7% 0.6% 27.4m 689 90.2% 2 flags
2026-09-05 138,731 88.0% 1.1% 37.8m 1,242 91.9% 2 flags
2026-09-04 348,984 87.2% 1.9% 2.2h 3,523 91.8% OK
2026-09-03 312,144 85.9% 1.9% 3.6h 3,392 92.4% OK
2026-09-02 343,296 83.9% 2.0% 2.7h 3,218 91.9% OK
2026-09-01 317,586 84.7% 1.6% 2.4h 3,147 91.7% OK
2026-08-31 250,842 90.7% 1.3% 1.5h 2,618 92.1% OK
2026-08-30 79,801 92.0% 0.7% 38.7m 496 89.3% 2 flags
2026-08-29 110,098 93.1% 1.1% 36.5m 1,098 91.1% 2 flags
2026-08-28 116,620 79.1% 1.7% 7.2h 1,146 91.9% 3 flags
2026-08-27 291,757 88.2% 1.7% 1.7h 3,218 92.2% OK
2026-08-26 350,536 90.4% 1.7% 2.7h 3,407 91.7% OK
2026-08-25 300,383 86.7% 1.2% 1.3h 3,081 92.2% OK
2026-08-24 294,153 89.7% 1.5% 1.8h 2,799 92.6% OK
2026-08-23 119,008 85.6% 0.3% 38.4m 1,140 89.9% 3 flags
2026-08-22 219,373 85.5% 0.6% 1.8h 2,311 93.3% OK
2026-08-21 335,015 85.5% 1.4% 1.3h 3,350 92.0% OK
2026-08-20 326,110 85.8% 1.5% 2.3h 3,306 91.1% OK
2026-08-19 292,715 86.7% 1.4% 2.8h 2,548 92.6% OK
2026-08-18 287,242 89.6% 1.6% 2.9h 3,352 92.3% OK
2026-08-17 251,343 93.5% 1.0% 1.7h 2,541 91.9% OK
2026-08-16 105,140 94.0% 0.9% 42.4m 831 90.4% 2 flags
2026-08-15 115,656 90.3% 0.8% 48.3m 939 90.5% 2 flags
2026-08-14 259,746 88.4% 1.7% 7.8h 3,166 91.9% OK
2026-08-13 331,283 80.5% 1.3% 10.9h 3,044 91.9% 2 flags
2026-08-12 323,102 87.3% 1.3% 11.8h 3,196 92.0% 1 flag
2026-08-11 279,710 92.7% 1.5% 2.8h 2,887 91.8% OK
2026-08-10 282,144 93.8% 1.0% 3.2h 3,013 91.7% OK
2026-08-09 89,219 95.8% 0.6% 1.0h 740 90.8% 2 flags
2026-08-08 149,149 84.9% 0.9% 1.7h 1,463 91.7% 2 flags
2026-08-07 320,157 87.1% 1.6% 3.3h 3,613 92.4% OK
2026-08-06 318,246 81.4% 1.3% 6.8h 3,129 91.7% 1 flag
2026-08-05 251,744 88.0% 1.5% 3.3h 2,495 90.8% OK
2026-08-04 302,106 85.4% 1.3% 6.0h 2,990 90.0% OK
2026-08-03 271,680 92.2% 1.2% 5.1h 2,974 91.2% OK
2026-08-02 115,959 93.6% 0.5% 1.3h 617 90.4% 2 flags
2026-08-01 105,101 91.4% 0.5% 2.1h 673 90.0% 3 flags
2026-07-31 280,433 87.6% 1.3% 5.7h 3,131 91.1% OK
2026-07-30 267,496 91.4% 1.1% 6.0h 3,070 91.0% OK
2026-07-29 290,139 91.3% 1.4% 3.3h 3,130 91.4% OK
2026-07-28 257,889 87.0% 1.6% 3.1h 2,869 90.8% OK
2026-07-27 268,027 88.7% 1.0% 2.5h 2,901 91.0% OK
2026-07-26 123,446 93.2% 0.6% 1.2h 1,239 90.6% OK
2026-07-25 83,494 92.4% 0.6% 2.4h 543 87.1% 2 flags
2026-07-24 230,603 89.9% 1.6% 1.5h 2,488 91.1% OK
2026-07-23 233,282 86.2% 1.4% 2.0h 2,347 91.7% 1 flag
2026-07-22 245,106 91.0% 1.6% 1.0h 2,769 91.5% OK
2026-07-21 278,123 90.3% 1.8% 4.3h 3,389 91.3% OK
2026-07-20 228,099 91.0% 1.5% 3.5h 2,393 91.3% OK
2026-07-19 86,558 92.8% 0.9% 49.0m 580 88.9% 2 flags