1 Background

Artificial intelligence (AI) is rapidly transforming the field of orthopaedic surgery, offering innovative solutions that extend beyond traditional diagnostic and treatment paradigms. AI-based tools, particularly large language models (LLMs) such as ChatGPT, Gemini, and Claude, have evolved into sophisticated multimodal systems capable of interpreting complex medical data, including both textual and visual inputs.¹ In orthopaedics, AI applications range from diagnosing fractures with sensitivity rates as high as 92.3 %, to predicting congenital hip dysplasia, meniscus tears, and osteoarthritis.^2–5 However, the application and reliability of AI-driven models, beyond ChatGPT, in orthopaedic education and clinical practice remain underexplored. There is a lack of global consensus on the efficacy of AI in clinical orthopaedic applications. For instance, while some studies highlight the success of major LLMs in triaging common sports medicine complaints of the knee joint, others criticize their ability to identify suitable candidates for surgical intervention.^6–8 This inconsistency underscores a gap in understanding the true utility of LLMs in orthopaedics, particularly those beyond ChatGPT.

The present study aims to fill this gap by conducting an evaluation and comparison of current state-of-the-art LLMs, including Claude-3-sonnet, GPT-4o, Gemini-1.5, and Meta's LLaMA Vision-Instruct models, using standardized questions from the Orthopaedic In-Training Examination (OITE) question bank. This analysis focuses not only on overall accuracy but also on crucial capabilities such as interpreting radiographic images, evaluating performance across subspecialties and assessing response consistency and confidence levels. This study also aims to compare the performance of AI model responses to gender-related questions to detect potential gender bias. Furthermore, unlike prior studies that primarily assessed the performance of a single popular LLM—ChatGPT—our study aims to compare the performances of all major LLMs, including Gemini, Claude, ChatGPT, and LLaMA, to one another and to the standard average performance of orthopaedic residents throughout various intervals of training.

2 Methods

3 Results

3.3

3.3 Performance by discipline

Analysis of model performance by discipline revealed a variation by subject area across all models (ANOVA, F = 9.29, p < 0.001). A strong positive correlation (Pearsons’ r = 0.998, p < 0.001) was observed between models' overall performance and their discipline-specific performance, indicating that models maintaining high general accuracy consistently performed well across subfields. All models had the best performance on questions categorized in the foundational topic of basic science and the worst performance on questions tagged in the trauma subfield (Fig. 1).

Close

Fig. 1 Heatmap of Model Performance Stratified by Discipline: Heatmap of model performance across medical disciplines, with accuracy scores ranging from 0.40 (light) to 0.80 (dark).

Export to PPT

3.4

3.4 Performance by image vs No image

An analysis of questions that contained images such as radiographs versus questions that contained text only revealed statistically significant lower performance on image-based questions compared to non-image questions across all models. The average decrease in performance was 10.9 % ± 1.6 % (t = 17.77, p < 0.001). GPT-4o maintained the highest accuracy in both categories (78.7 % no-image, 68.4 % with-image), while the 11B LLaMA model showed the lowest performance (49.7 % no-image, 41.0 % with-image). (Fig. 2).

Close

Fig. 2 Comparison of Model Performance on Questions with and without Images: Bars depict models' performance on questions stratified by image vs. no-image. Error bars indicate standard error of measurement.

Export to PPT

3.5

3.5 Confidence self-assessment

Additional analysis was performed to assess methods in which a user may be able to ensure accurate model responses, such as a maximum self-assessed confidence score or an agreement of responses across three independent trials (triplicate agreement). The first analysis was done using the confidence scores provided by every model for each question. All models were able to output confidence scores that were associated with accuracy (chi-square test, p < 0.001), except for LLaMA-11B (p = 0.6359) and Gemini-Flash (p = 0.2607). All of these models had improved accuracy in question responses when they reported a maximal confidence of 4/4 compared with overall accuracy (Z-test, all p < 0.001). However, it was noted that models tended to report higher confidence uniformly, with at least 95 % of responses returning scores of 3 or 4 in each model, and only 1 model recording a confidence score of 1 at all (Fig. 3).

Close

Fig. 3 Model Accuracy vs Confidence Level: Lines depict the relationship between model confidence scores (1–4) and accuracy (%). Point sizes indicate sample sizes (500–2500 questions).

Export to PPT

3.6

3.6 Combined strategy

The second set of analyses investigated triplicate agreement as a way for users to evaluate their level of confidence in the model's response (Fig. 4). A significant proportion of cases failed to produce triplicate agreement: 44.8 % for LLaMA 11B, 29.8 % for LLaMA 90B, 23.7 % for Claude-Sonnet, 6.8 % for Gemini-Flash, 5.9 % for Gemini-Pro, 5.3 % for GPT-4o-mini, and 3.0 % for GPT-4o. There was a strong positive correlation between overall model performance and the proportion of queries that resulted in a correct triplicate agreement (r = 0.914, p = 0.004). Furthermore, when there is a triplicate agreement, a higher accuracy than baseline is seen in all models (test of proportions, all p < 0.001). The largest increase was seen in LLaMA 90B (60.3 %–70.2 %) (Fig. 5).

Close

Fig. 4 Model Consistency and Accuracy: Bars represent distribution of model responses across three repeated prompts showing correct answers (0–3) and consistency patterns (colored bars). Purple bars indicate identical responses, the lighter green shows two matching responses, and dark green represents completely different responses across attempts.

Export to PPT

Close

Fig. 5 Model Performance by Subgroup. Bars depict model accuracy in scenarios of maximal confidence (confidence = 4.0), consistency of responses across three trials (triplicate agreement), or in scenarios of both occurring in tandem. Red dashed line indicates overall model accuracy.

Export to PPT

On the other hand, a significant proportion of queries were observed to have incorrect triplicate agreement (25.1 % for LLaMA 11B, 20.9 % LLaMA 90B, 16.9 % for Claude-Sonnet, 25.3 % for GPT-4o, 41.9 % for GPT-4o-mini, 43.5 % for Gemini-Flash, and 34.2 % for Gemini-Pro). These results represent "fixed false beliefs," which means that the LLM consistently produced the same wrong answer across multiple attempts (Fig. 4).

Finally, we evaluated the combinations of these factors (selected model, self-assessed confidence, and triplicate agreement) in identifying cases where the model output can be considered more reliable. The combination of these 3 factors successfully identifies high-performing subgroups of queries in GPT-4o and Claude-Sonnet, with accuracies of 83.4 % (n = 1592) and 79.6 % (n = 1899) respectively. In all models, triplicate agreement was observed to result in more accuracy individually than maximal confidence. However, a significant interaction between the confidence score and triplicate agreement was observed in all models (all interaction coefficients positive, all p < 0.001). The positive interaction coefficients indicate that high confidence combined with triplicate agreement had a synergistic effect on accuracy, indicating that the impact was greater than what would be expected from simply adding their individual effects (Fig. 5).

3.7

3.7 Female vs. male

The analysis of gender biases in model responses revealed that all models performed worse in questions with female patients, except for LLaMA-11B. Gemini-1.5-flash, Claude-3-sonnet, and Meta's Llama-3.2–90B all showed significant gender effects, with statistically significant worse performances in female-oriented questions (Fig. 6).

Close

Fig. 6 Model Performance Stratified by Gender: Bars depict accuracy on questions related to male (blue) or female (pink) patients. Asterisks (*) indicate statistically significant gender performance differences.

Export to PPT

4 Discussion

The primary objective of this study was to evaluate the foundational orthopaedic knowledge of multiple large language models (LLMs) as represented by their performance on Orthopaedic In-Training Examination (OITE) multiple-choice questions. However, it is important to emphasize that orthopaedic clinical decision-making is inherently complex, context-rich, and extends beyond what can be captured solely through MCQ-based assessments. Thus, while these results offer insight into the foundational knowledge base and educational potential of these models, they do not directly equate to clinical competence or real-world decision-making ability.

Our findings indicate significant variability in model performance, with OpenAI's GPT-4o emerging as the top performer, with Meta's Llama-3.2-11B-Vision-Instruct demonstrating the lowest accuracy. This variation may be attributed to differences in training data quality, model architecture, and the extent to which each model incorporates domain-specific medical literature.⁹ Additionally, OpenAI models likely benefited from reinforcement learning with human feedback (RLHF),¹⁰ which may have fine-tuned their responses to align more closely with medical reasoning and guidelines. Furthermore, local models such as LLaMA were trained on fewer parameters which may negatively impact performance across a broad range of clinical scenarios such as those presented by the OITE practice question set.

A key observation from our analysis is that all models performed significantly worse on image-based questions compared to text-based ones, highlighting the challenges of processing complex visual data. Orthopaedics is a visually intensive specialty that relies heavily on interpreting radiographic and clinical images, requiring an advanced understanding of spatial relationships and subtle abnormalities.^11,12 The poorer performance in this area may be due to a lack of sufficient annotated imaging data in the models' training datasets, limitations in their visual recognition capabilities compared to human perception, and the complexity of orthopaedic images that often require contextual clinical correlation.¹³ Additionally, LLMs may struggle with differences in image quality, minute variations in positioning, and the presence of artifacts that can obscure critical diagnostic features. Thus, the observed 10.9 % decrease in accuracy for image-based questions may be attributed to multiple factors and future work should systematically investigate these factors, ideally incorporating standardized imaging datasets and direct comparisons to expert human interpretation.

Across various orthopaedic subdisciplines, our results reveal that model performance is highest in basic sciences, while notable gaps persist in more complex areas such as trauma and pediatrics. This discrepancy may stem from several factors, including the broader availability of structured and standardized information on basic sciences, which is more commonly found in textbooks and research articles that these models are trained on.¹⁴ In contrast, trauma and pediatric orthopaedics involve more variability in presentation, complex decision-making based on evolving clinical scenarios, and fewer standardized guidelines, making it more challenging for models to generalize effectively.^15,16 Moreover, the dynamic nature of trauma cases,¹⁷ which often require individualized management approaches based on patient-specific factors, presents a unique challenge that AI models may not yet be equipped to handle.

Model confidence levels exhibited a general positive association with accuracy; however, no consistent trend was observed across different models. While higher self-appraised confidence often coincided with correct responses, inconsistencies in confidence reporting were noted, particularly with models that tended to overestimate their capabilities. Several potential factors could contribute to this phenomenon, including overfitting to certain types of data,¹⁸ limitations in the models' calibration algorithms,¹⁹ and a lack of proper validation datasets that reflect real-world clinical complexity.²⁰ Interestingly, none of the models, except LLaMA 90b, reported confidence levels below 2.0, suggesting a tendency to present responses with unwarranted certainty. This overconfidence may be influenced by the models’ training process, which often prioritizes fluency and coherence over accuracy, potentially leading to the generation of plausible-sounding but incorrect answers.^21–23

Our findings also highlight the importance of response consistency, with triplicate agreement being a stronger predictor of accuracy than confidence levels alone. This suggests that repeated agreement in responses may serve as a more reliable indicator of accuracy compared to self-appraised confidence. The improved accuracy with repeated responses may indicate that certain questions align well with the model's training data, while questions producing inconsistent responses may highlight areas where models lack sufficient understanding or have conflicting information within their knowledge base. However, this finding highlights the risk of models reinforcing incorrect information if a bias exists within the training data or if the same incorrect response is consistently generated.

Notably, our analysis revealed a synergistic interaction between confidence scores and triplicate agreement across all models, where their combined effect on accuracy exceeded the sum of their individual contributions. This effect suggests complex interactions between model confidence and response consistency that warrant further investigation. One possible mechanism is that high confidence combined with consistent responses indicates questions that fall within well-defined, thoroughly trained knowledge domains, while the absence of either factor may signal edge cases or areas of incomplete learning. However, the concerning prevalence of "fixed false beliefs" across all models highlights potential systematic errors in model knowledge and training methods. These consistent incorrect responses likely stem from biases or errors in training data that become deeply embedded in the model's knowledge representation. The variation in fixed false belief rates across models suggests that architectural improvements and refined training methodologies may help mitigate this issue. Future research should explore whether these reliability indicators remain stable across different task types and knowledge domains, whether they can be effectively leveraged in deployment settings to automatically flag responses requiring human verification, and how to better detect and correct fixed false beliefs during both training and inference.

A key finding of this analysis is the detection of gender bias within AI-generated responses. Questions pertaining to male patients yielded higher accuracy rates compared to those involving female patients in Gemini, Claude, and Llama 90b. The complexity of female-specific injury patterns, such as differences in bone density, hormonal influences, and anatomical variations, may contribute to the reduced accuracy in female-related cases, as well as potential biases in the training data available to these models.^24–27 The bias could also stem from a historical underrepresentation of women in clinical trials and research studies,²⁸ leading to a lack of sufficient data for AI models to learn from. Addressing such biases will be crucial for ensuring the equitable application of AI in orthopaedic education and practice. However, given the use of a keyword-based classification method and lack of expert validation or difficulty adjustment, these findings should be interpreted as exploratory rather than conclusive evidence of bias.

The qualitative analysis of model errors provided important insights into the limitations of current LLM capabilities. Errors frequently stemmed from insufficient understanding of complex clinical contexts and challenges in accurately interpreting subtle radiographic findings. These findings underscore that despite high numerical accuracy in foundational knowledge assessments, significant gaps remain in nuanced clinical judgment and image interpretation skills. Future research should include detailed qualitative error analysis to systematically identify and address specific weaknesses in LLM clinical reasoning.

While our study provides valuable insights into the capabilities and shortcomings of current LLMs in orthopaedics, several limitations must be acknowledged. The first is the absence of expert orthopaedic surgeon review of model-generated explanations or answer selections. As a result, we were unable to evaluate the clinical plausibility or educational quality of the responses beyond answer accuracy. Future studies should incorporate blinded expert review to assess the explanatory depth, clinical reasoning, and potential utility of LLM outputs in educational or decision-support settings.

A further limitation of this study is the potential for data leakage due to the public availability of the OITE question bank, despite the existence of a paywall. Since large language models are trained on extensive public datasets, it is possible that models had prior exposure to some of these questions, potentially inflating their accuracy. This represents an important confounder that must be considered when interpreting findings. Future studies aiming to provide more robust benchmarks should employ non-public, clinically validated question sets to minimize the risk of data leakage.

Importantly, our comparison of LLM performance to orthopaedic residents' scores must be interpreted cautiously. The percentile comparison used in this study is strictly limited to accuracy on standardized multiple-choice questions and does not reflect real-world clinical decision-making skills, contextual judgment, or patient-management abilities. The OITE is designed to assess resident knowledge, not clinical acumen or decision-making in real-world scenarios. Therefore, while our findings provide a benchmark for LLM performance on standardized educational material, they should not be interpreted as indicative of clinical competence or readiness for deployment in practice settings. Future studies should assess LLM performance using real patient cases, vignettes, and clinician-adjudicated scenarios to enhance clinical validity.

In relation to the gender bias analysis, a limitation was its reliance on a keyword-matching approach to classify questions as male- or female-oriented. While this method enabled large-scale categorization, it may oversimplify complex clinical narratives and potentially misclassify questions involving nuanced gender-related contexts. Additionally, no clinical expert validation was performed to confirm these classifications. Future studies should incorporate expert adjudication of gender attribution and examine intersectional factors that may influence model bias.

Another notable limitation is the lack of rigorous methodology for assessing visual interpretation skills. The radiographic images included in our test set were not standardized for quality or complexity, and we did not explicitly verify the LLMs’ capability to perform visual feature extraction or conduct human benchmarking to calibrate image-question difficulty. Future research should incorporate standardized imaging datasets, human expert validation, and formal evaluations of LLM image-processing abilities to provide more robust and clinically meaningful assessments.

A final limitation is our method of using triplicate agreement (consistency of responses across three trials) as an indicator of reliability. Although our data show correlations between triplicate agreement and accuracy, we did not formally evaluate test-retest reliability or correlate this measure with human expert confidence. Future studies should explicitly validate this method against established reliability standards and expert clinician confidence to confirm its utility as a reliability metric.

Despite these limitations, this study serves as a foundational benchmark for evaluating the applicability of various LLMs and their comparative performance in clinical orthopaedics. Future research should focus on improving the accuracy and uniformity of AI-generated responses through enhanced model training on diverse, high-quality orthopaedic datasets. Incorporating multimodal learning approaches that integrate both text and image data more effectively may enhance models' performance on visually complex questions. Additionally, efforts to fine-tune models with expert-curated datasets and real-world clinical cases can enhance their relevance and reliability. Further investigation into bias mitigation strategies, such as dataset augmentation with underrepresented patient demographics and the use of explainable AI techniques, will be essential to ensure fairness and transparency in AI-driven orthopaedic education. Longitudinal studies assessing the impact of AI-assisted learning tools on orthopaedic training outcomes will also be valuable in understanding their real-world utility.

CRediT authorship contribution statement

Gnaneswar Chundi: involved in the, Conceptualization, and design of the study, Material preparation, data collection, and, Formal analysis, were carried out by, Writing – original draft. Abhiram Dawar: involved in the, Conceptualization, and design of the study, Material preparation, data collection, and, Formal analysis, were carried out by, Writing – original draft. Syed Sarwar: involved in the, Conceptualization, and design of the study, Material preparation, data collection, and, Formal analysis, were carried out by. Sanjiv Prasad: involved in the, Conceptualization, and design of the study, Material preparation, data collection, and, Formal analysis, were carried out by. Michael Vosbikian: involved in the, Conceptualization, and design of the study, provided, Supervision. Irfan Ahmed: involved in the, Conceptualization, and design of the study, provided, Supervision, All authors reviewed and approved the final manuscript.

Data availability statement

The datasets generated and analyzed during the current study are available from the corresponding author upon reasonable request. Due to the proprietary nature of the Orthopaedic In-Training Examination (OITE) question bank, the full dataset cannot be publicly shared.

Permission to reproduce material from other sources (if applicable)

N/A.

For clinical trials (if applicable)

N/A.

Level of evidence

Level V.

Ethics statement

Not applicable for this study.

Patient consent statement (if applicable)

Not applicable for this study.

Funding

No funding was provided for this study.