{"degradation_reason":"rolling 30d precision 0.041 < 0.6","degraded":true,"generated_at":"2026-09-19T07:51:47.499992+00:00","label_definition":"A country-day is a POSITIVE outcome iff a citable-censorship incident (incident_type != 'disruption' AND status NOT IN ('suspected','draft')) has first_seen within the 7-day target window. Tightened 2026-06-21: the outcome join previously matched ANY incident, which counted IODA disruption outages (~65% of the incidents table) and single-source 'suspected' rows as ground-truth censorship \u2014 inflating the observed/positive rate (e.g. the lowest reliability bin's observed rate fell ~0.34 -> ~0.18 after the fix). These honest numbers are lower than before BY DESIGN; the model no longer gets credit for predicting outages.","methodology_url":"https://voidly.ai/sentinel/methodology","notes":"prod_rolling reflects real alerts evaluated against observed incidents in the 7-day target window. training_holdout is the stratified holdout from the last training run \u2014 it sanity-checks the model but is not evidence of live performance. When prod_rolling.n_evaluated is small, defer to training_holdout. When it's large, prod_rolling is authoritative.","prod_rolling":{"accuracy":0.5411111111111111,"brier_score":0.314,"calibration_bins":[{"bin_high":0.1,"bin_low":0.0,"n":441,"observed_rate":0.2766,"predicted_mean":0.032},{"bin_high":0.2,"bin_low":0.1,"n":15,"observed_rate":0.0,"predicted_mean":0.1481},{"bin_high":0.3,"bin_low":0.2,"n":33,"observed_rate":0.1515,"predicted_mean":0.2704},{"bin_high":0.4,"bin_low":0.3,"n":66,"observed_rate":0.1364,"predicted_mean":0.3489},{"bin_high":0.5,"bin_low":0.4,"n":41,"observed_rate":0.0244,"predicted_mean":0.4538},{"bin_high":0.6,"bin_low":0.5,"n":62,"observed_rate":0.0484,"predicted_mean":0.5511},{"bin_high":0.7,"bin_low":0.6,"n":103,"observed_rate":0.0388,"predicted_mean":0.6508},{"bin_high":0.8,"bin_low":0.7,"n":114,"observed_rate":0.0088,"predicted_mean":0.7479},{"bin_high":0.9,"bin_low":0.8,"n":25,"observed_rate":0.0,"predicted_mean":0.8392},{"bin_high":1.0,"bin_low":0.9,"n":0,"observed_rate":null,"predicted_mean":null}],"calibration_mae":0.3834,"confusion":{"false_negative":133,"false_positive":280,"total":900,"true_negative":475,"true_positive":12},"n_evaluated":900,"per_country":{"BD":{"false_negative":24,"false_positive":0,"n":30,"precision":null,"recall":0.0,"true_negative":6,"true_positive":0},"BR":{"false_negative":0,"false_positive":20,"n":30,"precision":0.0,"recall":null,"true_negative":10,"true_positive":0},"BY":{"false_negative":0,"false_positive":27,"n":30,"precision":0.0,"recall":null,"true_negative":3,"true_positive":0},"CN":{"false_negative":0,"false_positive":0,"n":30,"precision":null,"recall":null,"true_negative":30,"true_positive":0},"CU":{"false_negative":0,"false_positive":19,"n":30,"precision":0.0,"recall":null,"true_negative":11,"true_positive":0},"EG":{"false_negative":0,"false_positive":29,"n":30,"precision":0.03333333333333333,"recall":1.0,"true_negative":0,"true_positive":1},"ER":{"false_negative":0,"false_positive":6,"n":30,"precision":0.0,"recall":null,"true_negative":24,"true_positive":0},"ET":{"false_negative":0,"false_positive":8,"n":30,"precision":0.0,"recall":null,"true_negative":22,"true_positive":0},"ID":{"false_negative":19,"false_positive":0,"n":30,"precision":1.0,"recall":0.05,"true_negative":10,"true_positive":1},"IN":{"false_negative":18,"false_positive":0,"n":30,"precision":null,"recall":0.0,"true_negative":12,"true_positive":0},"IR":{"false_negative":3,"false_positive":16,"n":30,"precision":0.058823529411764705,"recall":0.25,"true_negative":10,"true_positive":1},"KP":{"false_negative":0,"false_positive":5,"n":30,"precision":0.0,"recall":null,"true_negative":25,"true_positive":0},"KZ":{"false_negative":4,"false_positive":17,"n":30,"precision":0.22727272727272727,"recall":0.5555555555555556,"true_negative":4,"true_positive":5},"LB":{"false_negative":0,"false_positive":0,"n":30,"precision":null,"recall":null,"true_negative":30,"true_positive":0},"MM":{"false_negative":0,"false_positive":22,"n":30,"precision":0.0,"recall":null,"true_negative":8,"true_positive":0},"MY":{"false_negative":25,"false_positive":0,"n":30,"precision":1.0,"recall":0.038461538461538464,"true_negative":4,"true_positive":1},"NG":{"false_negative":0,"false_positive":3,"n":30,"precision":0.0,"recall":null,"true_negative":27,"true_positive":0},"NI":{"false_negative":0,"false_positive":1,"n":30,"precision":0.0,"recall":null,"true_negative":29,"true_positive":0},"PH":{"false_negative":0,"false_positive":8,"n":30,"precision":0.0,"recall":null,"true_negative":22,"true_positive":0},"PK":{"false_negative":0,"false_positive":25,"n":30,"precision":0.0,"recall":null,"true_negative":5,"true_positive":0},"RU":{"false_negative":0,"false_positive":4,"n":30,"precision":0.2,"recall":1.0,"true_negative":25,"true_positive":1},"SA":{"false_negative":0,"false_positive":24,"n":30,"precision":0.04,"recall":1.0,"true_negative":5,"true_positive":1},"SD":{"false_negative":0,"false_positive":10,"n":30,"precision":0.0,"recall":null,"true_negative":20,"true_positive":0},"SY":{"false_negative":0,"false_positive":26,"n":30,"precision":0.0,"recall":null,"true_negative":4,"true_positive":0},"TH":{"false_negative":9,"false_positive":0,"n":30,"precision":1.0,"recall":0.1,"true_negative":20,"true_positive":1},"TM":{"false_negative":0,"false_positive":0,"n":30,"precision":null,"recall":null,"true_negative":30,"true_positive":0},"TR":{"false_negative":16,"false_positive":0,"n":30,"precision":null,"recall":0.0,"true_negative":14,"true_positive":0},"UZ":{"false_negative":0,"false_positive":5,"n":30,"precision":0.0,"recall":null,"true_negative":25,"true_positive":0},"VE":{"false_negative":5,"false_positive":0,"n":30,"precision":null,"recall":0.0,"true_negative":25,"true_positive":0},"VN":{"false_negative":10,"false_positive":5,"n":30,"precision":0.0,"recall":0.0,"true_negative":15,"true_positive":0}},"precision":0.0410958904109589,"recall":0.08275862068965517,"window_days":30},"published_warning":"Stratified AUC overstates real-world performance by 40.7pp vs. time-based split. Do not cite the stratified number as a deployment figure; use the loco_median or the prod_rolling block once it populates.","schema":"voidly-sentinel-accuracy/v1","split_audit_complete":true,"training_holdout":{"as_of":"2026-09-13T02:10:51.286057","f1":0.8446771378708552,"positive_rate":0.1306179775280899,"precision":0.8231292517006803,"recall":0.8673835125448028,"roc_auc":0.9814062068739227,"samples":2136,"source":"training_holdout","split_loco":{"f1_median":0.5045655375552283,"n_countries":21,"n_countries_evaluated":20,"notes":"Leave-country-out cross-validation. For each country, train on all other countries + eval on that country. The median is a more honest generalization figure than stratified. Individual per-country numbers reveal which countries the model struggles on (novel crises).","per_country":{"BD":{"auc":0.8942434096727324,"f1":0.6964856230031949,"n_pos_test":149,"n_test":731,"precision":0.6646341463414634,"recall":0.7315436241610739,"threshold":0.05},"BY":{"auc":0.8807326561486926,"f1":0.5142857142857142,"n_pos_test":70,"n_test":731,"precision":0.42857142857142855,"recall":0.6428571428571429,"threshold":0.15000000000000002},"CN":{"auc":0.896660510926439,"f1":0.12598425196850394,"n_pos_test":9,"n_test":731,"precision":0.06779661016949153,"recall":0.8888888888888888,"threshold":0.37000000000000005},"CU":{"auc":0.6498998163299382,"f1":0.11363636363636363,"n_pos_test":53,"n_test":731,"precision":0.14285714285714285,"recall":0.09433962264150944,"threshold":0.09000000000000001},"EG":{"auc":0.8886254140319181,"f1":0.6808510638297872,"n_pos_test":164,"n_test":731,"precision":0.6037735849056604,"recall":0.7804878048780488,"threshold":0.07},"ID":{"auc":0.9041736942516659,"f1":0.6008583690987125,"n_pos_test":118,"n_test":731,"precision":0.6086956521739131,"recall":0.5932203389830508,"threshold":0.28},"IR":{"auc":0.7975979817786174,"f1":0.4948453608247423,"n_pos_test":142,"n_test":731,"precision":0.48322147651006714,"recall":0.5070422535211268,"threshold":0.05},"KP":{"auc":null,"f1":null,"n_pos_test":0,"n_test":731,"note":"no positives in train or test","precision":null,"recall":null},"KZ":{"auc":0.8653542466922749,"f1":0.46153846153846156,"n_pos_test":71,"n_test":731,"precision":0.3312883435582822,"recall":0.7605633802816901,"threshold":0.33},"MM":{"auc":0.9386248698497694,"f1":0.6285714285714286,"n_pos_test":83,"n_test":731,"precision":0.5196850393700787,"recall":0.7951807228915663,"threshold":0.44000000000000006},"MY":{"auc":0.8063289783679588,"f1":0.424,"n_pos_test":87,"n_test":731,"precision":0.32515337423312884,"recall":0.6091954022988506,"threshold":0.15000000000000002},"PK":{"auc":0.8678637684203158,"f1":0.5224489795918368,"n_pos_test":89,"n_test":731,"precision":0.41025641025641024,"recall":0.7191011235955056,"threshold":0.31000000000000005},"RU":{"auc":0.6916211167231717,"f1":0.425629290617849,"n_pos_test":149,"n_test":731,"precision":0.3229166666666667,"recall":0.6241610738255033,"threshold":0.05},"SA":{"auc":0.8407917200170723,"f1":0.43312101910828027,"n_pos_test":71,"n_test":731,"precision":0.3953488372093023,"recall":0.4788732394366197,"threshold":0.17000000000000004},"SY":{"auc":0.9341004907042643,"f1":0.4807692307692308,"n_pos_test":42,"n_test":731,"precision":0.4032258064516129,"recall":0.5952380952380952,"threshold":0.42000000000000004},"TH":{"auc":0.9284286428214464,"f1":0.5843621399176955,"n_pos_test":87,"n_test":731,"precision":0.4551282051282051,"recall":0.8160919540229885,"threshold":0.39000000000000007},"TM":{"auc":0.6048645320197044,"f1":0.0,"n_pos_test":35,"n_test":731,"precision":0.0,"recall":0.0,"threshold":0.5},"TR":{"auc":0.8672301886792453,"f1":0.5811320754716981,"n_pos_test":106,"n_test":731,"precision":0.48427672955974843,"recall":0.7264150943396226,"threshold":0.16000000000000003},"UZ":{"auc":0.7530194179811547,"f1":0.34408602150537637,"n_pos_test":57,"n_test":731,"precision":0.24806201550387597,"recall":0.5614035087719298,"threshold":0.05},"VE":{"auc":0.895512081710516,"f1":0.7457627118644068,"n_pos_test":184,"n_test":731,"precision":0.7764705882352941,"recall":0.717391304347826,"threshold":0.08000000000000002},"VN":{"auc":0.8740814299900697,"f1":0.5303867403314917,"n_pos_test":95,"n_test":731,"precision":0.5581395348837209,"recall":0.5052631578947369,"threshold":0.32000000000000006}},"precision_median":0.41941391941391937,"recall_median":0.6335091083413231,"roc_auc_max":0.9386248698497694,"roc_auc_median":0.8709725992051927,"roc_auc_min":0.6048645320197044,"split":"loco"},"split_stratified":{"f1":0.8446771378708552,"n_test":2136,"notes":"Stratified random 15% holdout. Inflates AUC via within-country temporal leakage. Publish alongside time-based as the training-level sanity check, NOT as the deployment figure.","positive_rate":0.1306179775280899,"precision":0.8231292517006803,"recall":0.8673835125448028,"roc_auc":0.9814062068739227,"split":"stratified","threshold":0.25000000000000006},"split_summary":{"delta_stratified_vs_time_pp":40.71,"generated_at":"2026-09-13T03:00:20.684816+00:00","loco_median_auc":0.8709725992051927,"loco_median_f1":0.5045655375552283,"schema":"voidly-sentinel-evaluation/v1","stratified_auc":0.9814062068739227,"stratified_f1":0.8446771378708552,"time_based_auc":0.5743240946140051,"time_based_f1":0.5009861932938856},"split_time":{"f1":0.5009861932938856,"n_test":2303,"notes":"Time-based split \u2014 train on data before T, test on data after T. This is the honest deployment-aligned figure. Treat as the Sentinel baseline when n_evaluated in prod is < 30.","positive_rate":0.3321754233608337,"precision":0.3346508563899868,"recall":0.996078431372549,"roc_auc":0.5743240946140051,"split":"time","test_date_range":["2026-05-27","2026-09-13"],"threshold":0.05,"train_date_range":["2024-09-13","2026-02-04"],"val_date_range":["2026-02-04","2026-05-27"]},"threshold":0.25000000000000006},"window_days":30}