{"explainer":"running_accountability compares each archived daily shutdown_risk flag (one per country per day) with the Access Now KeepItOn shutdown starts recorded in the 7 days after it. A prediction is scored only when its whole 7-day window ends on or before label_coverage_end, the last start date in the KeepItOn file; later predictions are counted in n_unscored_after_label_coverage and leave precision and recall empty. KeepItOn is published about once a year, so recent windows stay unscored until the next release. validated_backtest is the v4 trainer's in-sample, full-panel summary, not a test of forecasts.","generated_at":"2026-11-22T06:30:48.838727+00:00","running_accountability":{"all_time":{"flag_rate":0.0,"n_actual_shutdowns_following":0,"n_false_negatives":0,"n_false_positives":0,"n_flagged":0,"n_predictions":1,"n_true_positives":0,"n_unscored_after_label_coverage":7966,"precision":null,"recall":null},"first_prediction_date":"2025-09-26","kio_countries_unmapped":["Armenia","Azerbaijan","Bahrain","Bangladesh","Benin","Brazil","Burkina Faso","Burundi","Burundi; Kenya; Rwanda; Tanzania, United Republic of; Uganda","Comoros","Congo","Cuba","Ecuador","Egypt","El Salvador","Eritrea","Eswatini","Gabon","Israel","Kazakhstan","Laos; Myanmar","Liberia","Malawi","Mali","Mauritania","Mauritius","Mozambique","New Caledonia","Niger","Senegal","Sierra Leone","Somaliland","Sri Lanka","Suriname","Tajikistan","Tunisia","United Kingdom","Uzbekistan","Zambia","Zimbabwe"],"kio_rows_used":1385,"label_coverage_end":"2025-12-29","label_source":"Access Now KeepItOn STOP (CSV export)","last_30_days":{"n_predictions":0,"n_unscored_after_label_coverage":1567,"precision":null,"recall":null,"status":"no_outcome_labels_for_this_window"},"last_7_days":{"n_predictions":0,"n_unscored_after_label_coverage":401,"precision":null,"recall":null,"status":"no_outcome_labels_for_this_window"},"latest_prediction_date":"2026-11-21","n_distinct_countries":60},"schema":"voidly-shutdown-risk-accountability/v1","validated_backtest":{"features":["kio_hist_5y","kio_recent_7d"],"honest_caveats":["Real KeepItOn-trained predictor. Country-level only.","Differentiates BETWEEN countries (the bulk of the AUC) much more than WITHIN a country day-to-day.","Score grows over time as new shutdowns are documented; the model is most reliable for countries with established history.","City-level / sub-national shutdowns (~half of KeepItOn events) are not strongly predictable from country aggregates.","Three prior versions (HistGB, LR, isotonic w/ temporal CV) lost this 0.89 signal to overfitting / non-stationary clipping; v4 is the minimal scorer that preserves it."],"honest_validation":{"auc_full_panel":0.947235932113929,"combined":{"auc":0.947235932113929,"pr_auc":0.42477464914655233},"comment":"These are full-panel AUCs because the predictor is essentially a country base-rate that grows over time as new shutdowns are documented. The 0.89 expresses how well the score RANKS country-days by true future shutdown probability. The model differentiates BETWEEN countries far more than WITHIN a country.","method":"score formula evaluated on the full labeled panel (no fold splits \u2014 the feature is non-stationary; rolling-origin clipping pathology made v1/v2/v3 underperform their own input)","pr_auc_full_panel":0.42477464914655233,"raw_kio_hist_5y":{"auc":0.9465505880862569,"pr_auc":0.4155774705950059},"raw_kio_recent_7d":{"auc":0.756077992275944,"pr_auc":0.35006697563339184}},"model_kind":"isotonic_calibrated_score","n_countries":60,"n_train":106533,"recommended_threshold":0.20027080176665915,"score_formula":"combined_h5_plus_2x_r7","threshold_metrics":{"f1":0.5192209991532599,"precision":0.5319222761970853,"recall":0.5071121402580219},"train_positive_rate":0.028376183905456525,"trained_at":"2026-11-22T06:30:11.113790+00:00","version":"shutdown_risk_v4"}}
