P-136: External Validation of a Machine Learning Model for Rapid Prediction of M-Spike Values in Multiple Myeloma Using the HealthTree Database

Introduction: Multiple myeloma (MM), a malignant proliferation of plasma cells, often presents diagnostic challenges due to its complex clinical manifestations and variable patient outcomes. Despite advances in diagnostic methods, such as serum and urine protein electrophoresis to measure M-spike proteins, these techniques are often limited by delayed turnaround times which can postpone critical treatment decisions and lead to patient anxiety. Our previous study introduced a machine learning model utilizing a Random Forest algorithm to predict M-spike values rapidly and accurately using the same-day routine clinical and laboratory data from a single institute. Given the potential of this model, external validation is crucial to confirm its effectiveness and reliability across a broader, independent patient cohort.

Methods: We conducted a retrospective analysis using an external dataset comprising 619 MM patients from HealthTree Foundation. This dataset was tabulated with comprehensive clinical and laboratory parameters like those used in our original model. We employed the same Random Forest model to this new dataset to predict M-spike values. The model's performance was evaluated based on its predictive accuracy, measured through the coefficient of determination (R²) and mean squared error between the predicted and actual M-spike values. Statistical analyses were performed using the same software environment as in the initial study, ensuring consistency in evaluation metrics.

Results: The median M-spike value was 0.16 (range 0.01-2.5 gr/dL). The residual distribution of the RF model indicated that nearly all M-spike values determined using the 43 variables distributed equally on either side of zero. The weighted value of each of the 43 independent variables was determined by individually removing a variable from the ML algorithm and measuring its effect on the mean squared error (MSE). The model demonstrated a strong predictive capability with an R² of 0.779 in the external validation, with Root Mean Square Error (RMSE) 0.2619 indicating substantial agreement between predicted and observed M-spike values. The scatter plot of predicted versus actual values highlighted the model's accuracy across the range of M-spike measurements. Feature analysis from the external validation aligned closely with our initial findings, with key predictors such as serum total protein and various lagged M-spike values maintaining high importance. This consistency supports the robustness of the model across different patient groups.

Conclusions: The external validation of our Random Forest model confirms its potential as a reliable tool for rapid M-spike prediction in MM patients. This model can significantly enhance clinical decision-making by providing faster predictions than traditional methods. Future research should focus on continuous refinement of the model by integrating more diverse datasets and exploring real-time validation in clinical settings.