{"calibration": {"calibrated": true, "loco_validated": {"auc_cal": 0.9789, "auc_raw": 0.982, "brier_cal": 0.0428, "brier_raw": 0.0563, "ece_cal": 0.01, "ece_raw": 0.0974}, "method": "isotonic (sklearn), leakage-safe LOCO-validated", "n_samples": 4237, "note": "/v1/classifier/score returns calibrated_probability alongside the raw probability. Isotonic preserves ranking (AUC) while improving Brier + ECE; validated leave-one-country-out (fit on all OTHER countries, evaluate on the held country)."}, "evaluation": {"dataset": {"positive_samples": 1116, "total_samples": 4237, "unique_countries": 131}, "honest_evaluation": {"distribution_note": "The headline LOCO MEDIAN F1 (0.870) is dominated by many small-sample countries scoring a perfect 1.0 on a handful of days; the MEAN F1 (0.711) is the honest single number. Censorship-heavy, high-volume countries score materially lower (CN ~0.29 on n=95, BY ~0.21 on n=84, AZ ~0.11 on n=65). Cite the mean \u2014 or the specific per-country number \u2014 for hard countries, not the median. The model is CLEAN (no label leakage); this is a distribution caveat, not an accuracy retraction.", "examples_hard_high_volume_countries": {"AZ": 0.10810810810810811, "BY": 0.21052631578947367, "CN": 0.29411764705882354}, "headline_metric": "LOCO mean F1 across all countries", "loco_mean_f1": 0.7109360224005077, "n_countries_perfect_f1": 46, "n_well_sampled_countries_n_ge_30": 61, "well_sampled_mean_f1": 0.63}, "loco_mean_f1": 0.7109360224005077, "loco_median_auc": null, "loco_median_f1": 0.8695652173913043, "loco_n_countries": 127, "notes": ["HONESTY: forward-temporal holdout (train past, test future) gives AUC 0.669 / F1 0.474 vs the random-split AUC 0.895 / F1 0.725. v3.3 generalizes across COUNTRIES (LOCO F1 0.87) but DEGRADES across TIME (delta AUC -0.226) \u2014 which is why it is retrained weekly. Do not read the random-split number as forward-deployment accuracy. See /atlas/findings/classifier-v3.3-temporal-generalization-2026-05."], "per_country": {"AE": {"auc": 0.80625, "f1": 0.36363636363636365, "n": 86, "precision": 0.4, "recall": 0.3333333333333333}, "AF": {"auc": 0.9, "f1": 0.8, "n": 7, "precision": 0.6666666666666666, "recall": 1.0}, "AM": {"auc": 0.7916666666666667, "f1": 0.6666666666666666, "n": 18, "precision": 1.0, "recall": 0.5}, "AO": {"auc": null, "f1": 0.8571428571428571, "n": 4, "precision": 1.0, "recall": 0.75}, "AR": {"auc": 0.9714285714285714, "f1": 0.8888888888888888, "n": 34, "precision": 0.9230769230769231, "recall": 0.8571428571428571}, "AT": {"auc": null, "f1": 1.0, "n": 2, "precision": 1.0, "recall": 1.0}, "AU": {"auc": 1.0, "f1": 1.0, "n": 39, "precision": 1.0, "recall": 1.0}, "AZ": {"auc": 0.5085227272727273, "f1": 0.10810810810810811, "n": 65, "precision": 0.5, "recall": 0.06060606060606061}, "BA": {"auc": 1.0, "f1": 1.0, "n": 8, "precision": 1.0, "recall": 1.0}, "BD": {"auc": 0.8081797235023042, "f1": 0.5306122448979592, "n": 87, "precision": 0.7222222222222222, "recall": 0.41935483870967744}, "BE": {"auc": null, "f1": 1.0, "n": 2, "precision": 1.0, "recall": 1.0}, "BF": {"auc": null, "f1": 1.0, "n": 1, "precision": 1.0, "recall": 1.0}, "BG": {"auc": 0.9025974025974025, "f1": 0.9230769230769231, "n": 18, "precision": 1.0, "recall": 0.8571428571428571}, "BH": {"auc": 1.0, "f1": 0.5, "n": 41, "precision": 0.3333333333333333, "recall": 1.0}, "BJ": {"auc": null, "f1": 0.6666666666666666, "n": 2, "precision": 1.0, "recall": 0.5}, "BO": {"auc": null, "f1": 0.6666666666666666, "n": 2, "precision": 1.0, "recall": 0.5}, "BR": {"auc": 1.0, "f1": 1.0, "n": 40, "precision": 1.0, "recall": 1.0}, "BW": {"auc": 1.0, "f1": 0.0, "n": 5, "precision": 0.0, "recall": 0.0}, "BY": {"auc": 0.6237037037037036, "f1": 0.21052631578947367, "n": 84, "precision": 0.13793103448275862, "recall": 0.4444444444444444}, "BZ": {"auc": 1.0, "f1": 0.6666666666666666, "n": 2, "precision": 0.5, "recall": 1.0}, "CA": {"auc": 1.0, "f1": 1.0, "n": 40, "precision": 1.0, "recall": 1.0}, "CD": {"auc": 1.0, "f1": 1.0, "n": 2, "precision": 1.0, "recall": 1.0}, "CG": {"auc": 0.8666666666666667, "f1": 0.5, "n": 8, "precision": 1.0, "recall": 0.3333333333333333}, "CH": {"auc": null, "f1": 1.0, "n": 1, "precision": 1.0, "recall": 1.0}, "CL": {"auc": 1.0, "f1": 1.0, "n": 8, "precision": 1.0, "recall": 1.0}, "CM": {"auc": 1.0, "f1": 1.0, "n": 16, "precision": 1.0, "recall": 1.0}, "CN": {"auc": 0.7292768959435627, "f1": 0.29411764705882354, "n": 95, "precision": 0.25, "recall": 0.35714285714285715}, "CO": {"auc": 0.96875, "f1": 0.9523809523809523, "n": 43, "precision": 1.0, "recall": 0.9090909090909091}, "CR": {"auc": null, "f1": 1.0, "n": 2, "precision": 1.0, "recall": 1.0}, "CU": {"auc": 0.7428571428571428, "f1": 0.3684210526315789, "n": 52, "precision": 0.25, "recall": 0.7}, "CZ": {"auc": null, "f1": 1.0, "n": 1, "precision": 1.0, "recall": 1.0}, "DE": {"auc": 1.0, "f1": 1.0, "n": 39, "precision": 1.0, "recall": 1.0}, "DO": {"auc": 1.0, "f1": 0.8, "n": 6, "precision": 1.0, "recall": 0.6666666666666666}, "DZ": {"auc": 0.6893382352941176, "f1": 0.3157894736842105, "n": 81, "precision": 0.2857142857142857, "recall": 0.35294117647058826}, "EC": {"auc": 1.0, "f1": 1.0, "n": 2, "precision": 1.0, "recall": 1.0}, "EE": {"auc": null, "f1": 0.5, "n": 3, "precision": 1.0, "recall": 0.3333333333333333}, "EG": {"auc": 0.7709447415329769, "f1": 0.7256637168141593, "n": 95, "precision": 0.5942028985507246, "recall": 0.9318181818181818}, "ES": {"auc": 1.0, "f1": 1.0, "n": 40, "precision": 1.0, "recall": 1.0}, "ET": {"auc": 0.9044871794871794, "f1": 0.6756756756756757, "n": 86, "precision": 0.5208333333333334, "recall": 0.9615384615384616}, "FI": {"auc": null, "f1": 1.0, "n": 1, "precision": 1.0, "recall": 1.0}, "FR": {"auc": 1.0, "f1": 1.0, "n": 42, "precision": 1.0, "recall": 1.0}, "GA": {"auc": 1.0, "f1": 1.0, "n": 6, "precision": 1.0, "recall": 1.0}, "GB": {"auc": 0.9901960784313726, "f1": 0.9090909090909091, "n": 47, "precision": 0.9375, "recall": 0.8823529411764706}, "GE": {"auc": 1.0, "f1": 0.8888888888888888, "n": 38, "precision": 0.8, "recall": 1.0}, "GH": {"auc": 1.0, "f1": 1.0, "n": 11, "precision": 1.0, "recall": 1.0}, "GQ": {"auc": null, "f1": 1.0, "n": 1, "precision": 1.0, "recall": 1.0}, "GR": {"auc": 0.9999999999999999, "f1": 1.0, "n": 31, "precision": 1.0, "recall": 1.0}, "GT": {"auc": 1.0, "f1": 0.6666666666666666, "n": 3, "precision": 0.5, "recall": 1.0}, "HK": {"auc": 1.0, "f1": 0.8, "n": 13, "precision": 1.0, "recall": 0.6666666666666666}, "HN": {"auc": 0.9363636363636364, "f1": 0.7272727272727273, "n": 41, "precision": 0.7272727272727273, "recall": 0.7272727272727273}, "HR": {"auc": null, "f1": 1.0, "n": 1, "precision": 1.0, "recall": 1.0}, "HU": {"auc": null, "f1": 1.0, "n": 4, "precision": 1.0, "recall": 1.0}, "ID": {"auc": 0.8284900284900284, "f1": 0.7111111111111111, "n": 92, "precision": 0.8888888888888888, "recall": 0.5925925925925926}, "IE": {"auc": 1.0, "f1": 1.0, "n": 3, "precision": 1.0, "recall": 1.0}, "IL": {"auc": 1.0, "f1": 1.0, "n": 13, "precision": 1.0, "recall": 1.0}, "IN": {"auc": 0.8472222222222222, "f1": 0.6779661016949152, "n": 90, "precision": 0.8695652173913043, "recall": 0.5555555555555556}, "IQ": {"auc": 0.8524691358024692, "f1": 0.6984126984126984, "n": 87, "precision": 0.6111111111111112, "recall": 0.8148148148148148}, "IR": {"auc": 0.9352678571428571, "f1": 0.85, "n": 96, "precision": 0.85, "recall": 0.85}, "IT": {"auc": 0.9305555555555556, "f1": 0.9090909090909091, "n": 42, "precision": 1.0, "recall": 0.8333333333333334}, "JM": {"auc": null, "f1": 1.0, "n": 1, "precision": 1.0, "recall": 1.0}, "JO": {"auc": 0.5538461538461538, "f1": 0.1935483870967742, "n": 77, "precision": 0.15789473684210525, "recall": 0.25}, "JP": {"auc": 1.0, "f1": 1.0, "n": 37, "precision": 1.0, "recall": 1.0}, "KE": {"auc": 0.9965277777777778, "f1": 0.8888888888888888, "n": 41, "precision": 0.8888888888888888, "recall": 0.8888888888888888}, "KH": {"auc": 0.8255813953488372, "f1": 0.3333333333333333, "n": 47, "precision": 0.5, "recall": 0.25}, "KR": {"auc": 0.8020833333333333, "f1": 0.8, "n": 35, "precision": 1.0, "recall": 0.6666666666666666}, "KW": {"auc": 0.8636363636363636, "f1": 0.0, "n": 13, "precision": 0.0, "recall": 0.0}, "KZ": {"auc": 0.8237179487179487, "f1": 0.5, "n": 85, "precision": 0.37037037037037035, "recall": 0.7692307692307693}, "LB": {"auc": 0.8181818181818182, "f1": 0.6206896551724138, "n": 52, "precision": 0.5, "recall": 0.8181818181818182}, "LT": {"auc": null, "f1": 1.0, "n": 6, "precision": 1.0, "recall": 1.0}, "LY": {"auc": 0.75, "f1": 0.0, "n": 55, "precision": 0.0, "recall": 0.0}, "MA": {"auc": 0.6118421052631579, "f1": 0.15384615384615385, "n": 82, "precision": 0.1, "recall": 0.3333333333333333}, "MD": {"auc": null, "f1": 0.8888888888888888, "n": 5, "precision": 1.0, "recall": 0.8}, "ME": {"auc": 0.9, "f1": 0.6666666666666666, "n": 7, "precision": 1.0, "recall": 0.5}, "MK": {"auc": 1.0, "f1": 1.0, "n": 4, "precision": 1.0, "recall": 1.0}, "MM": {"auc": 0.7619387027797577, "f1": 0.5454545454545454, "n": 84, "precision": 0.5714285714285714, "recall": 0.5217391304347826}, "MT": {"auc": null, "f1": 0.0, "n": 1, "precision": 0.0, "recall": 0.0}, "MU": {"auc": null, "f1": 0.0, "n": 1, "precision": 0.0, "recall": 0.0}, "MX": {"auc": 1.0, "f1": 1.0, "n": 55, "precision": 1.0, "recall": 1.0}, "MY": {"auc": 0.7466666666666667, "f1": 0.4, "n": 83, "precision": 0.42857142857142855, "recall": 0.375}, "MZ": {"auc": 1.0, "f1": 0.8888888888888888, "n": 6, "precision": 1.0, "recall": 0.8}, "NA": {"auc": null, "f1": 0.0, "n": 1, "precision": 0.0, "recall": 0.0}, "NE": {"auc": null, "f1": 1.0, "n": 1, "precision": 1.0, "recall": 1.0}, "NG": {"auc": 0.9691075514874142, "f1": 0.9041095890410958, "n": 61, "precision": 0.9428571428571428, "recall": 0.868421052631579}, "NI": {"auc": null, "f1": 1.0, "n": 6, "precision": 1.0, "recall": 1.0}, "NL": {"auc": 1.0, "f1": 0.875, "n": 10, "precision": 1.0, "recall": 0.7777777777777778}, "NO": {"auc": null, "f1": 1.0, "n": 1, "precision": 1.0, "recall": 1.0}, "NP": {"auc": 0.8579881656804733, "f1": 0.782608695652174, "n": 26, "precision": 0.9, "recall": 0.6923076923076923}, "NZ": {"auc": null, "f1": 1.0, "n": 2, "precision": 1.0, "recall": 1.0}, "OM": {"auc": 0.9438775510204083, "f1": 0.2857142857142857, "n": 53, "precision": 0.3333333333333333, "recall": 0.25}, "PA": {"auc": 1.0, "f1": 0.8888888888888888, "n": 12, "precision": 1.0, "recall": 0.8}, "PE": {"auc": null, "f1": 1.0, "n": 1, "precision": 1.0, "recall": 1.0}, "PH": {"auc": 0.9471458773784356, "f1": 0.8695652173913043, "n": 54, "precision": 0.8333333333333334, "recall": 0.9090909090909091}, "PK": {"auc": 0.6553571428571429, "f1": 0.4827586206896552, "n": 83, "precision": 0.6086956521739131, "recall": 0.4}, "PL": {"auc": 1.0, "f1": 1.0, "n": 31, "precision": 1.0, "recall": 1.0}, "PS": {"auc": null, "f1": 0.0, "n": 1, "precision": 0.0, "recall": 0.0}, "PT": {"auc": 1.0, "f1": 1.0, "n": 35, "precision": 1.0, "recall": 1.0}, "PY": {"auc": 1.0, "f1": 1.0, "n": 7, "precision": 1.0, "recall": 1.0}, "QA": {"auc": 0.788888888888889, "f1": 0.36363636363636365, "n": 36, "precision": 0.4, "recall": 0.3333333333333333}, "RO": {"auc": 0.8333333333333334, "f1": 0.875, "n": 23, "precision": 1.0, "recall": 0.7777777777777778}, "RS": {"auc": 1.0, "f1": 1.0, "n": 24, "precision": 1.0, "recall": 1.0}, "RU": {"auc": 0.7615925058548009, "f1": 0.6133333333333333, "n": 96, "precision": 0.575, "recall": 0.6571428571428571}, "SA": {"auc": 0.8783783783783784, "f1": 0.5185185185185185, "n": 84, "precision": 0.4117647058823529, "recall": 0.7}, "SC": {"auc": null, "f1": 1.0, "n": 1, "precision": 1.0, "recall": 1.0}, "SE": {"auc": 1.0, "f1": 1.0, "n": 2, "precision": 1.0, "recall": 1.0}, "SG": {"auc": 0.6157407407407407, "f1": 0.1, "n": 78, "precision": 0.07142857142857142, "recall": 0.16666666666666666}, "SI": {"auc": 0.9444444444444444, "f1": 0.875, "n": 13, "precision": 1.0, "recall": 0.7777777777777778}, "SK": {"auc": null, "f1": 1.0, "n": 1, "precision": 1.0, "recall": 1.0}, "SY": {"auc": 0.9221835075493612, "f1": 0.5333333333333333, "n": 62, "precision": 0.8888888888888888, "recall": 0.38095238095238093}, "TG": {"auc": 1.0, "f1": 1.0, "n": 8, "precision": 1.0, "recall": 1.0}, "TH": {"auc": 0.6564195298372514, "f1": 0.125, "n": 86, "precision": 0.1111111111111111, "recall": 0.14285714285714285}, "TN": {"auc": 0.703125, "f1": 0.3333333333333333, "n": 20, "precision": 0.5, "recall": 0.25}, "TR": {"auc": 0.7525, "f1": 0.35294117647058826, "n": 91, "precision": 0.23076923076923078, "recall": 0.75}, "TT": {"auc": 0.9587191358024691, "f1": 0.9484536082474226, "n": 75, "precision": 0.9387755102040817, "recall": 0.9583333333333334}, "TW": {"auc": 0.6351351351351351, "f1": 0.6666666666666666, "n": 39, "precision": 1.0, "recall": 0.5}, "TZ": {"auc": 1.0, "f1": 0.5, "n": 4, "precision": 1.0, "recall": 0.3333333333333333}, "UA": {"auc": 0.9928728070175439, "f1": 0.9387755102040817, "n": 62, "precision": 0.92, "recall": 0.9583333333333334}, "UG": {"auc": 1.0, "f1": 0.0, "n": 8, "precision": 0.0, "recall": 0.0}, "US": {"auc": 0.989795918367347, "f1": 0.7777777777777778, "n": 63, "precision": 0.6363636363636364, "recall": 1.0}, "UY": {"auc": 0.8571428571428572, "f1": 0.0, "n": 8, "precision": 0.0, "recall": 0.0}, "UZ": {"auc": 0.7097902097902098, "f1": 0.16, "n": 74, "precision": 0.6666666666666666, "recall": 0.09090909090909091}, "VE": {"auc": 0.9285387081997253, "f1": 0.9, "n": 96, "precision": 0.8852459016393442, "recall": 0.9152542372881356}, "VI": {"auc": null, "f1": 1.0, "n": 1, "precision": 1.0, "recall": 1.0}, "VN": {"auc": 0.5561224489795917, "f1": 0.3, "n": 84, "precision": 0.5, "recall": 0.21428571428571427}, "YE": {"auc": 0.7156862745098039, "f1": 0.0, "n": 53, "precision": 0.0, "recall": 0.0}, "ZA": {"auc": 0.7500000000000001, "f1": 0.8571428571428571, "n": 31, "precision": 1.0, "recall": 0.75}, "ZM": {"auc": null, "f1": 1.0, "n": 1, "precision": 1.0, "recall": 1.0}, "ZW": {"auc": 0.9038461538461539, "f1": 0.7619047619047619, "n": 17, "precision": 1.0, "recall": 0.6153846153846154}}, "stratified_auc": 0.8990661990023243, "stratified_f1": 0.7289360595519687}, "feature_names": ["anomaly_rate", "measurement_count", "spike_magnitude", "day_of_week", "month", "is_weekend", "rate_count_interaction", "probe_block_rate", "probe_node_count", "probe_avg_confidence", "probe_agreement", "rate_spike_interaction", "high_evidence", "neighbor_block_rate_7d", "neighbor_incident_count_7d", "neighbor_max_anomaly_7d"], "honest_caveat": {"confidence_is": "stratified cross-validation F1, NOT a calibrated per-prediction probability", "correct_use": "Cross-country censorship-risk classification. Per-country accuracy varies widely \u2014 16 MENA / former-Soviet countries (OM, UZ, TN, LY, YE, JO, MA, \u2026) regress 5-29pp on sparse neighbor-pair overlap. See evaluation.honest_evaluation + evaluation.per_country for the full distribution.", "headline_metric": "LOCO median F1 0.87 (cross-country); honest MEAN F1 0.71; well-sampled (n>=30 countries) mean ~0.63", "model_status": "CLEAN \u2014 no label leakage (ML_LEAKAGE_AUDIT.md); this caveat is about the score distribution, not leakage.", "training_label_composition": "794 of 1,116 positive training labels (71.1%) are country-days whose only incident is an IODA `disruption` row \u2014 network outages, not confirmed censorship. The same class was excluded from forecast labels in 2026-05 as ~94% noise. So this model is substantially trained to detect DISRUPTION, and its F1 should be read as such. Corpus is frozen at 2026-05-21; relabelling is a pending decision, not an oversight."}, "loaded": true, "metrics": {}, "model_type": "GradientBoostingClassifier", "path": "[internal path]", "schema": "voidly-classifier-info/v1", "trained_at": "2026-05-21T03:01:46.793987+00:00", "version": "v3.3"}