P-137: Optimizing Feature Selection for Large Language Models in AI-Assisted Clinical Assessment of Multiple Myeloma

Introduction: Large language models (LLMs) hold the potential to transform the healthcare industry by alleviating healthcare burdens, increasing accessibility in underserved areas, and providing multilingual support. Armed with extensive medical knowledge and natural language processing capabilities, these models require well-curated data inputs, or prompt engineering, for accurate diagnosis and personalized treatment planning. Multiple myeloma (MM), a complex hematological malignancy characterized by the proliferation of malignant plasma cells in the bone marrow, poses significant challenges due to its multisystemic impact. Implementing LLMs for MM clinical assessment necessitates identifying the most relevant features that reflect disease dynamics and patient status during each visit. This study utilizes a machine learning (ML) approach to select the most significant features for developing effective prompts for LLMs in the clinical assessment of MM patients.

Methods: The study analyzed 1,472 clinical observations from patients with MM, focusing on 43 clinical and laboratory variables to predict same-day M-spike values. A Random Forest (RF) algorithm, known for its robustness in handling nonlinear relationships and high-dimensional data, was employed. The dataset was split into a training set (80%) and a test set (20%) to validate the model. Bootstrapping was used to generate 500 data sets, constructing an ensemble of regression trees whose results were aggregated. The importance of each variable was assessed by examining the impact of its inclusion or exclusion on the model's mean squared error (MSE).

Results: The RF model demonstrated that nearly all M-spike values, determined using the 43 variables, were symmetrically distributed around zero residuals. Key variables identified as most influential included the first and second lagged M-spike values, serum total protein, serum IgG, serum IgM, and serum IgA. The RF model's predictions showed high correlation with laboratory-measured M-spike values, with Pearson and Spearman correlation coefficients of 0.96 and 0.91, respectively. Feature selection models were then compared to evaluate the minimal variable set required for accurate M-spike prediction. Five models (A-E) were developed, ranging from using all 43 variables to using only the most critical predictors. The Pearson correlation coefficients and root mean square error (RMSE) for these models were as follows: Model A (0.96, 0.21), Model B (0.96, 0.19), Model C (0.96, 0.19), Model D (0.95, 0.22), and Model E (0.95, 0.22).

Conclusions: This study highlights the importance of accurate feature selection in developing prompts for LLMs to assess MM. By identifying a curated set of variables most correlated with disease volume, we present potential list of parameters for prompt engineering to provide focused and comprehensive assessments. Future research should aim to refine these models further and explore their application in real-time clinical settings.