1 Introduction

Imaging such as X-Ray, Computed Tomography (CT), and Magnetic Resonance Imaging (MRI) are all invaluable forms for evaluating possible neoplasms within the context of bone tumors. Information such as size, extent of tumor, and likelihood of metastasis all can be obtained through imaging. However, these imaging techniques, which are commonly prescribed in the current American healthcare system, can be expensive and are linked to some of the unnecessary costs related to healthcare.

In 2018, Musculoskeletal Tumor Society (MSTS) and the American Academy of Orthopaedic Surgeons (AAOS) released specific guidelines on determining the most effective imaging modalities, catered to specific patients.³ These guidelines were created specifically to assist non-specialists when treating a bone or soft-tissue lesion of unknown origins.¹ The goal of these guidelines was to provide information regarding the urgency of and prior steps needed for referral to an orthopedic oncologist, while also limiting waste of patient's financial resources.⁴

Chat Generative Pre-Trained Transformer (ChatGPT) is an artificial intelligence large language model which generates data-based responses, and has been implemented experimentally within various fields, including healthcare.⁵,² The various advantages of ChatGPT, its ease of use, simple and easy to understand language, cannot overshadow its disadvantages. Limited specificity and specialization within medical knowledge, bias due to the data it models, and its oversimplification of complex medical topics.⁶ The utility of ChatGPT as a tool for medical provider use must be studied and evaluated, to help ensure that these biases and issues do not arise.

Thus, due to the accessible nature of ChatGPT and its emergence as a potential tool for medical evaluation, our objectives were to determine how closely ChatGPT recommendations align with MSTS guidelines for imaging bone and soft tissue lesions prior to orthopedic oncology referral. We also investigated if ChatGPT clearly and accurately explained these MSTS guidelines to the user.

2 Methods

This study is a cross-sectional analysis of ChatGPT's concordance with MSTS 2018 guidelines. Musculoskeletal Tumor Society's 2018 guidelines for use of imaging prior to referral to a musculoskeletal oncologist were used as the primary guideline. Questions were developed by the authors to assess ChatGPT's agreement with these guidelines. ChatGPT-3.5, developed by OpenAI, was the software used for this study.² A profile was generated specifically to pose questions to ChatGPT. Following each question and response, a new window was created within the ChatGPT software. If a question was misinterpreted by ChatGPT, a reworded version of the question was asked to provide ChatGPT with an opportunity to appropriately respond to the question.

The answers from ChatGPT were compared with MSTS guidelines and evaluated for concordance. A scoring system was developed based on previous literature which evaluated concordance of ChatGPT responses to current healthcare guidelines.⁷ Accuracy was determined by comparing the ChatGPT response with the MSTS guideline and checking for contradictory statements. After determining the accuracy of the ChatGPT response, three subcategories were evaluated to further explain differences between the MSTS guidelines and ChatGPT responses: over-conclusiveness, supplementary, and incompleteness (Fig. 1). The subcategories are further explained below.1.Accuracy: Is the ChatGPT response accurate with respect to the MSTS guidelines?aIf YES, the ChatGPT response did not contradict the MSTS guideline.bIf NO, the ChatGPT response contradicted the MSTS guideline.2.Over-conclusiveness: If the MSTS guidelines concluded that there was insufficient evidence to provide a recommendation, did ChatGPT provide one?aIf YES, ChatGPT made a recommendation while the MSTS guidelines did not provide a recommendation.bIf NO, either the MSTS guidelines provided a recommendation or both the MSTS guidelines and ChatGPT failed to provide a recommendation.3.Supplementary: Did ChatGPT include additional information relevant to the question which the MSTS guidelines did not specify?aIf YES, ChatGPT included significant additional information such as references to peer-reviewed articles or further explanations that were not included in the MSTS guidelines.bIf NO, ChatGPT did not contribute additional information relevant to the question.4.Incompleteness: If the ChatGPT response was accurate, did ChatGPT omit any relevant details which the MSTS guidelines included?aIf YES, ChatGPT failed to provide relevant information that was included in the MSTS guideline.bIf NO, the MSTS guideline did not contribute additional information that was not captured by ChatGPT.

Close

Fig. 1 Scoring system to evaluate ChatGPT alignment with MSTS guidelines.

Export to PPT

Each ChatGPT response was evaluated by two independent reviewers (K.E. and S.F.). Any disagreements were settled by two senior reviewers (A.D. and I.H.). The accuracy, over-conclusiveness, supplementary, and incompleteness of responses were compared using chi-square test. Statistical significance was set to a p-value of 0.05.

3 Results

In total, 12 guidelines were obtained from MSTS recommendations for imaging prior to orthopaedic oncology referral. We generated 14 questions to assess concordance with the 12 guidelines (Supplemental Fig. 1). Results showed alignment between the accuracy of ChatGPT's responses and the 12 guidelines. Of the 14 questions posed to ChatGPT, 10 were deemed accurate to the guidelines (71%). These responses did not contradict the MSTS guidelines or provide additional or incomplete information.

Of the 14 questions asked, responses to 13 questions (93%) were deemed to be overconclusive or to contain supplementary information (p < 0.05). When questioned about MRI being able to distinguish benign and malignant lesions, ChatGPT generated supplementary information regarding detailed contrast enhancement patterns (well beyond the scope of the MSTS guideline for the same question), as well as some additional comments on tumor vascularity. Similarly, 9 responses (64%) were deemed incomplete when compared to MSTS guidelines (p < 0.05) (Fig. 2). When questioned about ultrasound as an apt imaging modality to distinguish between benign and malignant soft tissue tumors, the MSTS guidelines were explicit that ultrasound was far more effective at making this distinction in small (<5cm) rather than large (>5cm) deep soft tissue tumors. ChatGPT was far less explicit, stating instead that “ultimately, the diagnosis of soft tissue tumors often requires a combination of imaging studies, clinical evaluation, and sometimes biopsy for definitive diagnosis” and that “other imaging modalities such as MRI or CT scans may be used in conjunction with ultrasound to provide a more comprehensive assessment.”

Close

Fig. 2 Results from a Pearson's Chi-Squared Goodness of Fit Test, an evaluation of ChatGPT Responses to MSTS Guidelines.

Export to PPT

4 Discussion

The value of imaging a musculoskeletal tumor prior to orthopedic oncology referral has long been reported in clinical literature. More recent work, however, has indicated a growing tendency towards excessive and unnecessary use of advanced imaging techniques, even prior to cancer specialist referral.^8–10 To clarify the standards for using such advanced techniques, the MSTS generated evidence-based practice guidelines that proposed to “provide guidance regarding imaging options and delivery in musculoskeletal tumors of unknown biological significance.”³ Among the aims of this project were the reduction in the use of “costly and unnecessary imaging”, the surety of conducting the correct studies for given clinical indications, and lastly, the proper and efficient use of advanced imaging techniques in diagnosing bone and soft tissue tumors.³ Its intended use, the group stated, was for all members of the medical field who encountered a potential orthopedic malignancy and desired a standard set of recommendations during early management of said malignancy.

The use of artificial intelligence (AI), specifically the deep learning-based language model ChatGPT, has become ubiquitous in many fields. Contrarily the medical field has largely restricted its use, for reasons ranging from credibility to added legal complexities.¹¹ However, when used specifically as a search engine for direct replies to queries, analysis of existing literature, and even as a virtual assistant for “patient monitoring in case of exacerbation of risk factors”, ChatGPT has shown promise.^11,12 Its ability to significantly reduce time burden when conducting tasks and its capacity to translate and explain patient data and diagnoses highlights the advantages that ChatGPT holds for the healthcare professional.¹³ In the field of orthopedic surgery specifically, a 2023 study showed that in the field of orthopedic surgery specifically, “AI-enabled ChatGPT has been adopted to perform several functions in orthopedics, such as helping in education, suggesting medical care, and individual case analysis in surgery.”¹⁴ In terms of its diagnostic use, a 2021 study by the Liu group found that AI was just as accurate in identifying tibial plateau fractures as orthopedic specialists.^15,16 ChatGPT has been minimally studied in orthopaedic oncology, but there is room for the literature to expand on this topic.¹⁷

In the specific setting of cancer diagnosis, these advantages are particularly appealing. In March 2024, when ChatGPT was asked for specific information regarding head, neck, liver, prostate, breast, lung, pancreas, colon and cervical cancers, its “responses were accurate, dependable, and aligned with the expertise of oncology professionals, especially for certain subtypes of cancers like head and neck and prostate cancers.”¹⁸ More recently, an article assessed the alignment of ChatGPT's responses to questions about bone and soft tissue tumor biology with the responses of orthopedic oncologists.¹⁷ The findings revealed a startling consistency in responses, underscoring our group's rationale for testing the alignment of ChatGPT's guidance with existing evidence-based standards (from MSTS) for bone and soft tissue tumors.

For our inquiry into the integrity of ChatGPT recommendations in comparison to the MSTS guidelines, this group generated 14 questions based on said guidelines, using the Mount Sinai's Cho group's 2023 analysis of ChatGPT recommendations as a guide.⁷ Upon inputting our questions into the search engine of ChatGPT-3.5 our results suggested a convincing alignment of ChatGPT's recommendations. Ideally, the language model's responses would have mirrored the MSTS imaging guidelines. Instead, much like the Cho group's results, we obtained reasonably accurate responses for 10 responses, yet simultaneous overconclusive or supplemental information for 13 responses, and 9 incomplete responses. This tendency for over-conclusion, additions and incompleteness may be negated by newer versions of the same language model, which have been found to be notably more conservative in its responses.⁷

In addition to the above-mentioned tendencies, our research shows ChatGPT is highly descriptive and layman-friendly in describing imaging modalities. CT scans and MRI readings, while the bread and butter of most clinicians and non-specialists, are not always featured in everyday vocabulary. As seen in Fig. 3, ChatGPT clearly describes these imaging techniques in an accessible, accurate manner, elevating its advantage as a virtual assistant in supporting medical diagnoses.

Close

Fig. 3 ChatGPT window demonstrating an accurate response.

Export to PPT

On the contrary, ChatGPT provides a lack of rigor that some orthopedic oncology specialists might prefer in supporting their diagnoses or treatment strategies. Our findings demonstrate that an oncologist-led diagnosis can be supported by deep learning modeling, rather than a diagnosis defined by ChatGPT itself. Furthermore, ChatGPT provides a lack of transparency with the sources it uses to produce its responses unless otherwise asked; upon positing the question ‘what are your sources for this information?’ following the response seen in Fig. 3, ChatGPT initially declined to give specific sources or databases, instead choosing general guidelines and websites, as well as an assortment of textbooks and manuals. Upon more insistent questioning for specificity, however, appropriate cited works were offered as sources (Supplemental Fig. 2).

The general validity of the MSTS guidelines as a viable tool for musculoskeletal tumor assessment has recently been tested by the Milano group, who assessed the guidelines for use in Italy on both patient and specialist provider (orthopedic surgeon) sides.¹⁹ In the specific context of MSTS's imaging recommendations, no systematic review of its clinical use in diagnosing bone and soft tissue lesions has emerged, but such a review that assesses both patient and provider (i.e. subjective and objective) understanding of the imaging guidelines would be rewarding. In 2018, AAOS supported the original systematic literature review that detailed the MSTS imaging guidelines, generally stating that non-specialist orders for further studies beyond an MRI are not required prior to referral, and that radiographs were often enough for the non-specialist clinician.²⁰ These independent findings also support our ChatGPT-derived results, which is an admittedly evolving clinical modality that cannot yet standalone in its utility in this discipline, but whose recommendations also generally align with clinical opinion.

There are several limitations associated with our study. Firstly, the MSTS clinical guidelines are from 2018 and therefore may be outdated as further literature regarding this topic has been published after 2018. Similarly, ChatGPT-3.5 was used for all responses, while a newer version of ChatGPT, ChatGPT-4.0, was more recently released and encompasses more recent data. Finally, the ratings for each subcategory were in part subjective and based on author evaluation for concordance. Therefore, the true accuracy of ChatGPT's responses to MSTS guidelines cannot be quantified from this study. ChatGPT does not always provide specific answers and at times needs to be prompted for specificity and references. At times, the answers were generalized or accommodating multiple imaging techniques, inevitably leading to a degree of incompleteness that is a legitimate limitation of the AI model. Despite these limitations, we present the most readily available information on ChatGPT's concordance with MSTS guidelines for orthopaedic oncology imaging prior to referral.

5 Conclusions

The concordance between the accuracy of responses and the 2018 MSTS guidelines is convincing, suggesting that imaging methods recommended by MSTS to front-line practitioners are generally aligned with AI-based modeling recommendations. However, the clinical use of this software must be further investigated as the results demonstrate that ChatGPT often produces supplemental or incomplete information. While AI-based predictions in healthcare are an evolving modality, their value as diagnostic tools demand further rigorous research.

CRediT authorship contribution statement

Karen Ebenezer: Formal analysis, Investigation, Writing – original draft, Resources. Sarah Folk: Formal analysis, Investigation, Writing – original draft, Resources. Alexis M. Driscoll: Conceptualization, Methodology, Writing – review & editing, Resources. Isabel Herzog: Conceptualization, Methodology, Writing – review & editing. Kathleen Beebe: Conceptualization, Writing – review & editing, Supervision, Funding acquisition: none.

Informed consent

Informed consent was not necessary to be obtained for this study.

Ethical statement

Institutional Review Board Approval is not necessary for this non-human subjects review of public data.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.