Queue Forecasting Dashboard

Hour (UTC)	Pending	Started	Resolved	Completed	Failed
00:00	3,159	3,225	3,724	3,531	58
01:00	5,252	4,631	2,344	2,200	82
02:00	2,827	3,463	5,458	5,103	204
03:00	4,130	3,984	3,425	3,314	70
04:00	1,990	2,167	3,236	3,102	98
05:00	5,734	5,522	3,717	3,543	142
06:00	5,016	4,955	6,757	6,055	321
07:00	2,948	2,785	2,406	2,287	103
08:00	4,981	4,897	4,710	3,788	648
09:00	14,050	11,083	11,422	7,290	353
10:00	10,969	10,516	11,878	10,679	527
11:00	14,719	12,378	15,141	12,869	376
12:00	10,863	10,866	9,762	9,082	395
13:00	5,739	5,114	5,878	5,491	215

Table Sizes

Table	Rows	Newest	Age
queue_forecast_daily_health	141	2026-07-20 13:14:40Z	40m ago
queue_forecast_run_predictions	9,671,527	2026-07-20 13:44:15Z	10m ago
queue_forecast_task_runs	12,627,076	2026-07-20 13:44:15Z	10m ago
queue_forecast_tasks	12,411,957	2026-07-20 13:44:11Z	10m ago
queue_forecast_worker_counts	7,093,496	2026-07-20 13:40:38Z	14m ago
queue_forecast_worker_pools	722	2026-07-19 13:48:33Z	1.0d ago

Stream Freshness

Stream	Latest	Age
pending_at	2026-07-20 13:44:15Z	10m ago	OK
started_at	2026-07-20 13:44:15Z	10m ago	OK
resolved_at	2026-07-20 13:44:13Z	10m ago	OK
enriched_at	2026-07-20 13:44:11Z	10m ago	OK
worker_sample	2026-07-20 13:40:38Z	14m ago	OK
daily_health	2026-07-20 13:14:40Z	40m ago	OK

Ingestion Windows

Window	Pending	Started	Resolved
5m	2,680	1,532	439
15m	3,429	2,360	1,390
1h	7,835	7,506	8,360
6h	62,260	55,736	59,612
24h	125,911	118,214	122,548

Open Issues

Metric	Value
daily_health_rows_last_7d	7
unenriched_tasks	10
unresolved_runs	11,174
unresolved_runs_older_than_2h	7,696
worker_samples_last_30m	6

Training Results

Reason	Count (1h)
completed	7,822
failed	285
canceled	123
worker-shutdown	103
claim-expired	4
deadline-exceeded	2

trained 1.5d ago · as_of 2026-07-18 · lightgbm

train: 5,691,987 rows (2026-06-12..2026-07-12) · val: 8,718 · holdout: 1,168,397

	Model	Baseline	Delta
MAE	2.1m	2.3m	-9.9%
within-2x	92.8%	91.5%	—
p90 coverage	91.3%	—	—

Per-day holdout breakdown

Day	n	MAE	w/in 2x	p90 cov
2026-07-13	233,017	2.0m	93.4%	92.3%
2026-07-14	226,397	2.7m	92.7%	88.7%
2026-07-15	236,264	1.9m	92.7%	92.4%
2026-07-16	213,498	2.0m	92.3%	90.6%
2026-07-17	197,367	1.8m	92.6%	92.5%

trained 1.5d ago · as_of 2026-07-18 · lightgbm

train: 2,723,328 rows (2026-06-28..2026-07-12) · val: 8,859 · holdout: 1,189,019

	Model	Baseline	Delta
MAE	6.6m	7.2m	-7.7%
within-2x	61.2%	56.4%	—
p90 coverage	86.0%	—	—

Bucket	n	MAE	Base MAE	w/in 2x	Base w/in 2x
<1m	570,180	19.8s	26.3s	48.5%	44.0%
1-5m	287,568	1.2m	1.4m	77.2%	71.3%
5-30m	179,487	5.3m	5.8m	77.0%	74.1%
30m+	62,095	1.6h	1.7h	58.2%	49.8%

Per-day holdout breakdown

Day	n	MAE	w/in 2x	p90 cov
2026-07-13	228,113	10.0m	59.5%	83.4%
2026-07-14	226,397	8.4m	62.2%	86.7%
2026-07-15	233,955	4.2m	60.6%	87.1%
2026-07-16	213,498	4.7m	61.9%	86.9%
2026-07-17	197,367	5.7m	62.0%	85.9%

Walk-Forward Evaluation

Cohort	Baseline MAE	Model MAE	Δ MAE %	Model within-2x	Δ within-2x pp	p90 cov	Hold rows
2026-06-14	6.6m	7.1m	8.6%	60.3%	+6.6pp	86.6%	1,118,075
2026-06-15	5.9m	7.5m	27.5%	61.5%	+6.9pp	88.5%	957,318
2026-06-16	5.2m	5.6m	9.2%	61.7%	+7.5pp	88.4%	934,004
2026-06-17	4.9m	5.1m	3.4%	61.4%	+7.2pp	87.7%	955,808
2026-06-18	4.6m	4.1m	-11.0%	61.1%	+7.5pp	87.7%	973,736
2026-06-19	4.8m	4.5m	-6.1%	59.9%	+7.2pp	87.1%	1,143,310
2026-06-20	4.7m	4.3m	-8.3%	59.4%	+6.9pp	87.9%	1,233,221
2026-06-21	3.8m	3.6m	-5.7%	60.0%	+6.8pp	88.1%	1,083,974
2026-06-22	3.3m	3.0m	-8.7%	60.5%	+7.4pp	87.6%	912,098
2026-06-23	3.1m	2.8m	-8.8%	61.2%	+7.6pp	86.8%	881,216
2026-06-24	3.7m	3.0m	-18.7%	61.9%	+7.6pp	86.3%	849,901
2026-06-25	3.7m	2.9m	-20.1%	61.9%	+7.6pp	87.2%	875,168
2026-06-26	5.6m	4.4m	-21.7%	61.2%	+7.3pp	87.7%	1,007,592
2026-06-27	5.5m	4.4m	-21.3%	60.6%	+6.7pp	87.5%	1,148,457

Cohort	Baseline MAE	Model MAE	Δ MAE %	Model within-2x	Δ within-2x pp	p90 cov	Hold rows
2026-06-14	6.6m	5.6m	-15.3%	65.8%	+12.1pp	86.4%	1,118,075
2026-06-15	5.9m	5.6m	-5.6%	67.3%	+12.7pp	88.1%	957,318
2026-06-16	5.2m	4.2m	-17.9%	66.7%	+12.5pp	87.2%	934,004
2026-06-17	4.9m	3.8m	-22.0%	66.5%	+12.3pp	87.8%	955,808
2026-06-18	4.6m	3.5m	-24.6%	66.4%	+12.8pp	87.5%	973,736
2026-06-19	4.8m	3.9m	-17.8%	65.6%	+12.8pp	87.0%	1,143,310
2026-06-20	4.7m	4.0m	-14.2%	65.2%	+12.7pp	87.6%	1,233,221
2026-06-21	3.8m	3.3m	-14.7%	66.1%	+12.8pp	87.4%	1,083,974
2026-06-22	3.3m	2.7m	-19.0%	65.6%	+12.4pp	86.7%	912,098
2026-06-23	3.1m	2.5m	-18.7%	66.7%	+13.1pp	86.2%	881,216
2026-06-24	3.7m	2.6m	-28.6%	67.2%	+12.9pp	85.9%	849,901
2026-06-25	3.7m	2.6m	-30.0%	67.4%	+13.1pp	87.0%	875,168
2026-06-26	5.6m	3.7m	-34.1%	66.8%	+13.0pp	86.9%	1,007,592
2026-06-27	5.5m	3.7m	-33.6%	66.5%	+12.6pp	87.1%	1,148,457

Cohort	Baseline MAE	Model MAE	Δ MAE %	Model within-2x	Δ within-2x pp	p90 cov	Hold rows
2026-06-14	6.6m	5.4m	-17.3%	66.3%	+12.6pp	86.4%	1,118,075
2026-06-15	5.9m	5.1m	-13.8%	67.8%	+13.1pp	87.9%	957,318
2026-06-16	5.2m	4.2m	-18.3%	67.1%	+12.9pp	87.9%	934,004
2026-06-17	4.9m	4.5m	-7.7%	66.5%	+12.3pp	87.5%	955,808
2026-06-18	4.6m	3.3m	-27.0%	66.5%	+12.9pp	87.4%	973,736
2026-06-19	4.8m	4.0m	-16.7%	65.8%	+13.0pp	87.3%	1,143,310
2026-06-20	4.7m	3.9m	-16.4%	65.5%	+13.0pp	87.1%	1,233,221
2026-06-21	3.8m	3.3m	-13.2%	66.2%	+12.9pp	87.0%	1,083,974
2026-06-22	3.3m	2.7m	-17.2%	66.5%	+13.3pp	86.7%	912,098
2026-06-23	3.1m	2.5m	-19.2%	66.8%	+13.2pp	85.9%	881,216
2026-06-24	3.7m	2.6m	-28.0%	67.5%	+13.1pp	85.7%	849,901
2026-06-25	3.7m	2.5m	-32.0%	67.5%	+13.1pp	86.4%	875,168
2026-06-26	5.6m	3.7m	-33.8%	66.9%	+13.0pp	87.4%	1,007,592
2026-06-27	5.5m	3.6m	-35.0%	67.0%	+13.1pp	87.0%	1,148,457

Cohort	Baseline MAE	Model MAE	Δ MAE %	Model within-2x	Δ within-2x pp	p90 cov	Hold rows
2026-06-14	6.6m	6.7m	1.4%	58.5%	+4.9pp	88.2%	1,118,075
2026-06-15	5.9m	7.7m	30.6%	59.6%	+5.0pp	89.2%	957,318
2026-06-16	5.2m	5.0m	-2.7%	59.5%	+5.3pp	88.9%	934,004
2026-06-17	4.9m	5.2m	6.7%	59.4%	+5.2pp	88.8%	955,808
2026-06-18	4.6m	4.1m	-10.0%	59.2%	+5.6pp	87.9%	973,736
2026-06-19	4.8m	4.5m	-5.7%	57.7%	+4.9pp	87.4%	1,143,310
2026-06-20	4.7m	4.4m	-4.9%	57.4%	+4.9pp	88.4%	1,233,221
2026-06-21	3.8m	3.7m	-4.1%	58.1%	+4.8pp	87.8%	1,083,974
2026-06-22	3.3m	3.1m	-4.6%	58.6%	+5.4pp	87.8%	912,098
2026-06-23	3.1m	2.9m	-5.8%	59.2%	+5.6pp	87.0%	881,216
2026-06-24	3.7m	3.1m	-15.2%	59.8%	+5.5pp	86.9%	849,901
2026-06-25	3.7m	2.9m	-19.4%	59.9%	+5.6pp	87.8%	875,168
2026-06-26	5.6m	4.5m	-20.5%	58.9%	+5.0pp	87.9%	1,007,592
2026-06-27	5.5m	4.3m	-22.6%	58.6%	+4.7pp	87.6%	1,148,457

Cohort	Baseline MAE	Model MAE	Δ MAE %	Model within-2x	Δ within-2x pp	p90 cov	Hold rows
2026-06-14	6.6m	5.4m	-17.6%	66.2%	+12.6pp	86.5%	1,118,075
2026-06-15	5.9m	5.0m	-14.4%	67.5%	+12.9pp	87.9%	957,318
2026-06-16	5.2m	4.3m	-16.1%	66.9%	+12.7pp	87.3%	934,004
2026-06-17	4.9m	4.2m	-14.8%	66.5%	+12.3pp	88.0%	955,808
2026-06-18	4.6m	3.8m	-17.2%	66.5%	+12.9pp	87.3%	973,736
2026-06-19	4.8m	4.1m	-14.2%	65.9%	+13.1pp	87.3%	1,143,310
2026-06-20	4.7m	4.0m	-14.3%	65.4%	+12.9pp	86.8%	1,233,221
2026-06-21	3.8m	3.4m	-12.3%	66.2%	+12.9pp	87.6%	1,083,974
2026-06-22	3.3m	2.8m	-14.0%	66.3%	+13.1pp	86.5%	912,098
2026-06-23	3.1m	2.4m	-21.6%	66.9%	+13.3pp	86.1%	881,216
2026-06-24	3.7m	2.7m	-26.2%	67.5%	+13.2pp	85.9%	849,901
2026-06-25	3.7m	2.6m	-29.0%	67.6%	+13.3pp	86.5%	875,168
2026-06-26	5.6m	3.7m	-34.5%	67.2%	+13.3pp	87.1%	1,007,592
2026-06-27	5.5m	3.7m	-33.0%	66.9%	+13.1pp	87.3%	1,148,457

Date	Total	Comp%	Exc%	Wait p99	Cap p50	Util	Anomaly
2026-07-19	86,558	92.8%	0.9%	49.0m	580	88.9%	2 flags
2026-07-18	86,222	92.6%	0.9%	1.6h	599	90.3%	2 flags
2026-07-17	223,913	88.1%	1.9%	4.1h	2,115	90.9%	OK
2026-07-16	244,534	87.3%	2.0%	2.6h	2,403	91.3%	OK
2026-07-15	268,542	88.0%	1.6%	2.1h	2,730	91.2%	OK
2026-07-14	253,194	89.4%	1.9%	5.1h	3,270	91.8%	OK
2026-07-13	269,148	86.6%	1.7%	7.2h	4,678	91.6%	1 flag
2026-07-12	11,610	71.6%	1.7%	24.2m	544	90.5%	3 flags
2026-07-11	28,193	76.2%	1.1%	1.5h	158	83.7%	3 flags
2026-07-10	230,458	90.7%	1.9%	2.3h	2,264	91.1%	OK
2026-07-09	253,304	88.9%	2.1%	1.9h	2,945	91.3%	OK
2026-07-08	246,086	90.6%	1.7%	2.8h	2,473	91.4%	OK
2026-07-07	274,613	88.0%	1.7%	9.4h	3,189	91.7%	1 flag
2026-07-06	231,663	91.6%	1.4%	1.9h	2,392	91.1%	OK
2026-07-05	95,793	89.7%	0.8%	42.8m	742	90.1%	2 flags
2026-07-04	68,704	89.2%	0.6%	33.8m	457	89.1%	2 flags
2026-07-03	237,329	87.2%	1.3%	1.6h	2,330	90.8%	OK
2026-07-02	243,832	88.7%	1.6%	2.6h	3,040	87.9%	OK
2026-07-01	243,298	87.8%	1.7%	1.5h	2,954	86.4%	OK
2026-06-30	283,032	87.6%	1.7%	2.4h	3,959	87.3%	OK
2026-06-29	268,060	92.9%	1.7%	57.3m	3,912	87.0%	OK
2026-06-28	142,128	93.1%	0.7%	2.1h	1,764	88.4%	OK
2026-06-27	101,674	86.6%	0.9%	29.8m	840	87.8%	3 flags
2026-06-26	234,954	88.7%	1.9%	1.4h	3,127	86.4%	OK
2026-06-25	262,436	87.3%	1.8%	4.2h	3,807	84.9%	1 flag
2026-06-24	235,100	87.6%	1.7%	52.7m	2,839	85.1%	OK
2026-06-23	235,882	90.6%	2.0%	5.0h	3,361	84.6%	1 flag
2026-06-22	225,703	92.0%	1.6%	2.4h	3,298	86.0%	OK
2026-06-21	88,695	92.9%	1.0%	25.0m	683	86.7%	2 flags
2026-06-20	129,566	86.3%	0.9%	36.9m	1,305	84.5%	3 flags
2026-06-19	206,884	87.5%	1.7%	1.3h	2,083	85.9%	OK
2026-06-18	267,552	91.2%	2.0%	1.4h	4,020	86.2%	OK
2026-06-17	267,900	86.5%	1.8%	1.9h	3,741	85.8%	OK
2026-06-16	266,853	89.8%	1.7%	2.7h	3,467	85.8%	OK
2026-06-15	320,774	74.0%	4.2%	6.1h	4,217	86.7%	2 flags
2026-06-14	113,118	85.0%	5.0%	38.3m	994	85.3%	3 flags
2026-06-13	98,192	88.9%	1.1%	28.1m	1,038	89.2%	2 flags
2026-06-12	258,308	82.0%	2.2%	3.2h	3,150	85.8%	OK
2026-06-11	266,114	79.8%	2.9%	2.0h	3,788	85.3%	OK
2026-06-10	306,794	85.4%	2.6%	5.5h	3,991	85.7%	OK
2026-06-09	285,287	85.8%	2.2%	2.0h	3,822	85.3%	OK
2026-06-08	239,038	84.2%	1.6%	2.0h	3,112	85.7%	OK
2026-06-07	95,770	94.9%	1.0%	24.1m	879	87.4%	2 flags
2026-06-06	99,658	87.6%	1.5%	42.5m	954	86.1%	2 flags
2026-06-05	210,502	83.0%	2.1%	3.1h	2,236	86.9%	1 flag
2026-06-04	257,291	83.6%	2.0%	4.2h	3,213	84.3%	OK
2026-06-03	297,748	84.4%	2.2%	3.5h	4,027	84.8%	OK
2026-06-02	250,210	86.0%	2.1%	2.8h	3,512	85.1%	1 flag
2026-06-01	242,418	89.6%	1.8%	2.1h	3,530	85.8%	OK
2026-05-31	102,261	94.6%	1.0%	41.2m	1,014	87.2%	2 flags
2026-05-30	121,825	83.4%	1.1%	1.1h	1,208	86.8%	2 flags
2026-05-29	297,213	84.8%	2.0%	4.2h	3,671	86.6%	OK
2026-05-28	249,726	89.2%	2.1%	2.9h	3,517	80.7%	OK
2026-05-27	256,661	84.7%	1.5%	1.5h	3,753	74.2%	OK
2026-05-26	222,248	90.9%	1.8%	2.2h	3,673	78.3%	OK
2026-05-25	163,251	95.2%	1.0%	35.1m	2,318	81.2%	OK
2026-05-24	99,096	86.1%	0.7%	29.8m	865	83.2%	3 flags
2026-05-23	168,672	80.4%	0.9%	5.1h	2,351	82.2%	1 flag
2026-05-22	271,701	86.3%	1.7%	10.4h	3,574	78.0%	2 flags
2026-05-21	291,509	86.4%	1.5%	4.0h	4,610	75.1%	OK

Queue Forecasting Dashboard

Today (UTC)

Data Collection Health