Production forensic audit · immutable source review

MLB prediction system deep audit

Slate 20260928. This report separates the canonical final decision from the stochastic inning simulation, audits signed probability confidence, reconciles starters against the official schedule, and records the exact readiness of the adaptive learning and training loop.

0games audited

0winner conflicts

0score conflicts

0starter mismatches

0unresolved starters

INCOMPLETEslate artifacts

BLOCKEDoverall readiness

Executive conclusion

Canonical display rule: the public winner and score come from final_prediction. A disagreeing scoring_simulation remains visible for diagnosis but cannot overwrite the accepted winner.

Confidence rule: Signed confidence edge = home win probability − away win probability. Positive is home; negative is away. Winner probability remains separately available as pick_probability for calibration and learning.

Artifact safety: the source game JSON files were read only. Files mutated: 0.

Official slate completeness

Status: INCOMPLETE. Expected 0 official games and found 0 valid canonical artifacts.

Missing games: None.

Unexpected games: None.

Artifact parse failures: 0. The command returns a nonzero status for incomplete slates unless --allow-incomplete is explicit.

All games at a glance

Game	Canonical final	Diagnostic simulation	Signed edge	Starter status A/H

Final vs simulation contradictions

A score difference is diagnostic. A winner difference is a hard public-surface contradiction unless the display is synchronized to the final decision.

Fixes and verification

ID	Status	Summary	Verification
canonical_public_result	FIXED	Public winner and score now come from final_prediction.	The simulation remains preserved as diagnostic evidence and can no longer overwrite the accepted public decision.
signed_confidence_contract	FIXED	Confidence now means home_win_probability minus away_win_probability.	pick_probability and calibration_probability remain separate winner probabilities for learning and calibration.
signed_confidence_learning_separation	FIXED	Brier, calibration buckets, thresholds, and expected-hit math never consume the signed UI edge.	Away favorites use pick_probability/calibration_probability in the 0-1 probability domain while retaining a negative public confidence edge.
official_starter_precedence	FIXED_FOR_NEXT_SAFE_GENERATION	A current official MLB probable pitcher replaces a conflicting local starter.	The source and replacement reason are recorded; post-first-pitch artifacts are not silently rewritten.
starter_only_history	FIXED	Starter matchup quality excludes relief appearances.	Starter flags/pitching order are preferred, with a conservative innings fallback when role flags are absent.
starter_workload_weighting	FIXED	Starter quality is weighted by expected innings and the remaining bullpen share is neutral.	A starter-only factor is preserved for audit while the game receives the workload-adjusted factor.
hourly_source_refresh	FIXED	Long-running workers refresh schedule and pitcher caches when source revisions change.	Revision-aware cache keys cover daily starters, player stats, pitcher performance, recent rows, and matchup splits.
learning_artifact_merge	FIXED	Postgame learning reads root final/simulation fields together with nested game analysis.	Final-versus-simulation contradictions and signed probability edges enter the learning evidence packet.
dynamic_starter_learning_rows	FIXED	Daily and rolling learning rebuild starter rows for the actual selected dates.	The obsolete hard-coded August 1–3 starter report is no longer the normal pipeline input; each historical date resolves its own archived slate for the separate 120-game adaptive window.
official_settlement_safety	FIXED	Official actuals refresh until the slate is final and doubleheaders settle by game number.	In-progress starter lines are excluded, final cache payloads are reusable, and game 1/game 2 cannot overwrite or cross-pair their starters.
temporal_leakage_guard	FIXED	Adaptive results require timestamp proof that the artifact was generated before first pitch.	Missing, post-start, or overwritten artifacts are excluded from packets, ledgers, evaluations, and proposal sample counts.
adaptive_evidence_binding	FIXED	Promotion evidence is bound to evaluator mode and exact champion/candidate hashes.	Promotions reload a content-addressed system evaluation artifact and verify its hash; unsupported modes, unrelated metrics, partial normalized weight groups, and unimplemented auto-promotion fail closed.
starter_projection_evaluator	FIXED	Starter-only form variables now use frozen prospective replay and official starter outcomes.	Promotion is based only on starter ER/outs MAE paired by game, side, and MLB ID; generic game Brier evidence is rejected.
phase_aware_trace_storage	FIXED	Prediction traces are request-local, atomically saved, and versioned after first pitch.	Concurrent requests cannot share a tracer, same-second reruns receive unique revision names, and the canonical pregame trace is preserved.

Readiness checks

ID	Status	Evidence
report_contract	PASS	Audited 0 games and generated equivalent JSON/HTML outputs.
source_artifact_immutability	PASS	Mutated canonical source files: [].
official_schedule	BLOCKED	Official schedule unavailable: no rows
slate_artifact_completeness	BLOCKED	Expected 0 official games; found 0 valid artifacts; missing []; unexpected []; parse failures 0.
current_artifact_starter_identity	PASS	0 mismatches and 0 unresolved starter sides in immutable source artifacts.
immutable_source_public_contract	PASS	0 immutable source artifacts violate the signed confidence, winner/score, probability-direction, or probability-sum contract. They were audited, not rewritten.
current_generation_public_contract	PASS	Current generation code synchronizes public winner/score and computes confidence as home_win_probability - away_win_probability; immutable legacy artifacts remain visibly separate.
postgame_settlement_and_learning	PENDING	At least one game is not final. Run settlement/learning again after official final status.
adaptive_core_modes	PASS	Artifact replay, enhanced shadow, and isolated XGBoost training backtest are mode/hash bound and manually promoted.
starter_projection_metrics	PASS	Allowlisted starter prior/weight candidates use prospective frozen replay, official MLB-ID settlement, ER/outs MAE, bootstrap gates, exact hashes, manual promotion, and starter-aware rollback. Global/hitter variables remain excluded.

Artifact parsing and integrity

Artifact	Result	Detail
No artifact parsing errors.

Model provenance

The provenance for every game is retained inside its detail card. This exposes schema, training timestamp, row count, feature count, and publication gate rather than treating the current model as an unexplained black box.

Adaptive learning, review, and training context

Deep findings

20260819_STL@CIN

The better starter was recognized; the projected loss was not caused by rating him as worse.

Chase Burns is the stronger starter in the artifact, but the accepted team-level decision still favors St. Louis because season/recent team signals outweigh Cincinnati's starter edge. The diagnostic simulation does not project Burns being charged with every St. Louis run.

Matthew Liberatore profile: ERA N/A, WHIP N/A; diagnostic line N/A IP and None ER.
Chase Burns profile: ERA N/A, WHIP N/A; diagnostic line N/A IP and None ER.
Burns suppresses the away run environment (factor N/A); Liberatore raises Cincinnati's run environment (factor N/A).
Published-winner diagnostic signals include winner quality edge N/A, recent-form edge N/A, and pitcher edge N/A.
A 6 IP / 3 ER starter line is not a five-run collapse: the bullpen is responsible for the remaining simulated runs.

Action/readiness: Starter-only rows, official identity precedence, expected-innings weighting, and hourly cache refresh are corrected for future safe pregame generations. The immutable already-started artifact is audited, not rewritten.

20260819_WAS@TEX

The Washington 4–2 screen was a cross-stage display bug, not the accepted forecast.

The accepted decision is None with None as winner; the disagreeing diagnostic simulation is None with None as winner.

Home probability N/A, away probability N/A, signed home-minus-away confidence N/A.
The old page combined the simulation score/winner with the final-policy probabilities, which created the visible contradiction.
The local home starter was None; the official probable is None, so the source artifact is marked .

Action/readiness: The public contract is synchronized to final_prediction, and future pregame generation replaces the conflicting local starter with the official MLB probable. No post-first-pitch source JSON was overwritten.

20260819 full slate

The contradiction was systemic across display surfaces, while winner conflicts were concentrated.

0 of 0 artifacts have a final/simulation score difference and 0 have a winner difference.

Starter audit: 0 matches, 0 mismatches, and 0 unresolved sides.
Probability-sum violations: 0; immutable signed-confidence violations: 0.
Every game card retains both stages so an operator or LLM can diagnose the disagreement without granting the simulation publication authority.

Action/readiness: Regenerate future pregame artifacts with the corrected code; keep today's already-started source artifacts immutable and use this report as the correction/audit layer.

Connected execution flow

Hourly upstream files change; revision-aware loaders invalidate same-process schedule, player, pitcher, recent-form, and matchup caches.
Pregame orchestration reconciles each starter against the current official MLB game row and records source/provenance.
Starter-only historical quality and Savant quality are blended by expected starter innings; the bullpen owns the remaining game share.
The stable final-decision policy chooses one winner/score, then the public contract synchronizes every public field while retaining the simulation as diagnostic.
After official settlement, review_prediction_results joins actual scores to the immutable prediction and preserves signed edge plus pick_probability.
The daily and rolling postgame pipeline rebuilds starter comparisons for the selected dates; a separate 120-game adaptive evidence window prevents daily sample limits from disabling proposals.
The adaptive cycle ingests settled evidence, evaluates one bounded candidate, asks OpenAI then Claude for strict JSON only when eligible, and never lets LLM text directly mutate production.
Starter candidates replay frozen pregame form evidence and settle against separate append-only official starter events; XGBoost candidates use isolated chronological retraining.
A deterministic, mode/hash-bound evaluation must pass before manual promotion; approved changes retain exact rollback and starter-aware or game-aware post-promotion monitoring.

Known limits and safety holds

Games already past first pitch remain immutable. Corrections apply to the public audit layer and to the next safe pregame generation.
The full-pipeline evaluator is intentionally starter-only: global player-form, hitter, and unscoped pitcher parameters remain non-promotable until they receive their own outcome evaluator.
The repository has no resident scheduler. Deployment must invoke prediction refresh before games and settlement/learning after games become final.
Automatic promotion is intentionally unsupported. OpenAI/Claude propose bounded JSON; deterministic tests and an operator approve production activation.
An official schedule with no announced probable pitcher remains unresolved rather than being labeled verified.

Complete machine-readable operating context

{
  "data_directory": {
    "status": "untouched_by_this_audit",
    "policy": "The report reads output/games and the external official schedule only. It does not rewrite or enumerate data/."
  },
  "adaptive_readiness": {
    "artifact_replay": "operational",
    "enhanced_shadow": "operational",
    "model_training_backtest": "operational_with_manual_promotion",
    "full_pipeline_backtest": "operational_for_allowlisted_starter_projection_parameters_only",
    "provider_order": [
      "openai",
      "anthropic"
    ],
    "provider_output_authority": "proposal_only"
  },
  "confidence_semantics": {
    "display": "home_win_probability - away_win_probability",
    "calibration": "pick_probability / calibration_probability"
  }
}

Evidence sources

Official MLB Stats API schedule — game identity, current status, and probable-pitcher IDs/names
FOX Sports STL–CIN game page — Matthew Liberatore versus Chase Burns and the pregame market context
Baseball Savant — Matthew Liberatore — official pitcher identity and Statcast profile context
FOX Sports WAS–TEX game page — Cade Cavalli versus Kumar Rocker and the pregame market context