Bridging the Validity Gap: Mitigating Strategic Overfitting in Management Analytics through Advanced Predictive Metrics

Abstract

Purpose: In the age of data-driven management, leaders of organizations are relying more and more on structural models to predict the return on investments in knowledge. But there is a serious "validity gap": models with high explanatory power (R²) often don't accurately predict outcomes that weren't used to train the model, which can lead to strategic overfitting and decision risk. This study shows that advanced predictive metrics (Q² Predict, RMSE, and CVPAT) can be used to make diagnoses in Knowledge-Based Transformation Models (KBTM).

Approach: We use Partial Least Squares Structural Equation Modelling (PLS-SEM) along with PLS-Predict and Cross-Validated Predictive Ability Testing (CVPAT) on a dataset of 310 banking professionals in Ghana. We look at how well the model predicts outcomes that weren't used to train it across nine multivariate constructs, separating internal knowledge processes from external

performance outcomes.

Findings: show that there is a lot of variation at the construct level. Knowledge-intensive processes, like Knowledge Creation (Q² = 0.834), are much better at predicting outcomes than naive benchmarks (p < 0.001). On the other hand, performance outcomes (like Employee Performance, Q² = 0.191) don't do a good job of predicting what will happen, which means that the model is very uncertain.

Value: This study goes beyond traditional fit indices to create a set of rules for predictive validation in management analytics. We show that only using explanatory metrics hides "blind spots" in predicting performance. We suggest a framework for analytics maturity that encourages leaders to tell the difference between outcome variables that can change and process variables that can change before making strategic decisions.