Quantifying Representation-Induced Decision Drift in Reliability-Critical Machine Learning Systems

Machine learning models are increasingly deployed in reliability-critical workflows in manufacturing, sensing, and prognostics, where discrete operational decisions-such as maintenance triggers, quality holds, and alarm issuance-directly affect safety, cost, and asset availability. Validation practice in reliability engineering typically emphasizes predictive accuracy, discrimination metrics, and calibration performance. These metrics do not explicitly assess the stability of discrete decisions under alternative but informationally equivalent feature representations.

This paper introduces a quantitative metric, representationinduced decision drift, defined as the fraction of discrete operational decisions that change under deterministic preprocessing transformations while model class, training split, and hyperparameters remain fixed. The metric is evaluated on three reliability-relevant public datasets: SECOM semiconductor manufacturing quality data, Gas Sensor Array Drift data, and NASA C-MAPSS FD001 turbofan prognostics data. Results demonstrate domain-dependent drift ranging from near-zero levels (≤ 0.2%) to substantial instability exceeding 30%, including cases where predictive accuracy remains nearly unchanged.

Nonparametric bootstrap confidence intervals confirm statistical robustness under an i.i.d. test-set assumption, and guidance is provided for temporally correlated reliability data using block bootstrap variants. The proposed metric and accompanying audit procedure support reliability validation, change management, and governance in ML-enabled systems.