Machine learning models used in production settings are frequently retrained as new data becomes available to improve accuracy. However, this causes inconsistencies between the predictions provided by different versions of the model, a process known as prediction churn. This is undesirable for end users and organizations, as it might result in inconsistent outcomes. In this study, we compared and assessed two current methods for reducing prediction churn, Anchor and Distillation. We proposed and demonstrated that introducing extra unlabeled data into both methods might further reduce prediction churn, resulting in a novel methodology. The results suggest that employing the Anchor and Distillation approaches individually minimizes prediction churn, but our improved strategy, which includes more unlabeled data, produces even better consistency, translating to more reliable model predictions for our clients.
