Dealing with Bias in Artificial Intelligence Driven Psychotherapy Tools Among Cultural and Racial Populations

Psychotherapy as a tool for treating various mental and physical health disorders has long been established as an effective treatment modality for mental disorders in Western populations, demonstrating efficacy and long-term efficiency (Kim et al., 2019). However, some authors argue that Western approaches and models in psychotherapy may not be suitable for Black Africans (Shatte et al., 2019) and for other cultures. Others advocate for a return to traditional psychotherapeutic paradigms, suggesting that in Africa and other continents of the world, healing is not solely based on the individual alone, but between therapist and patient and also as a collective process involving the community. The traditional African system, for instance, has an inherent psychological treatment framework. For example, African Grief Therapy, evident in burial rites and ceremonies, and the ritual bathing of child soldiers for cleansing, facilitates the assumption of a new identity and coping with the past (Nwoye, 2000).

Currently, there is no doubt that available artificial intelligence (AI) technologies could automate some of the time and labor-intensive aspects of clinical practices and therapies, thereby improving the efficiency and accessibility of psychological services in both public and private sectors (Huang et al., 2019). It is conceivable that AI will eventually enable the development and implementation of autonomous psychotherapy-bots capable of providing fully automated psychological services and therapies (Collins et al., 2021). Kleiman and colleagues (2017) found that using ecological momentary assessment data factors such as hopelessness and loneliness also correlated with suicidal ideation for clients over time. This knowledge paves the way for idiographic, real-time assessments that can inform truly personalized interventions (DeRubeis, 2019; Weisz et al., 2019). Another advantage of artificial intelligence in psychotherapy is its potential to enhance our identification of clinical populations and improve our ability to match interventions to the subgroups most likely to benefit (Kretzschmar et al., 2019).

Research in the application of AI in psychotherapy across cultures is advancing (Kim et al., 2019; McGreevey et al., 2020) with software being developed for language recognition and processing. Also, AI and machine learning – a subset of AI that focuses on developing algorithms and statistical models that allow computers to perform tasks without explicit instructions -are increasingly utilized in mental healthcare and psychotherapeutic intervention (Torous et al., 2020), however, their reliability across cultures and varied populations is limited. One emerging technology is the Generative Pretrained Transformer (GPT-4.5) AI model, including agents and chatbots, which are often integrated into mobile applications and various medical devices. These tools help in providing mental health support or solutions, frequently addressing depression, anxiety, and other disorders (Adamopoulou & Moussiades, 2020).

Clinicians and psychotherapists have categorized AI tasks and processes into three areas: mechanical, thinking, and feeling (Huang et al., 2019). Huang and colleagues (2019) posited that AI could readily handle mechanical (robotics) and thinking tasks (processing, analyzing, and interpreting data), but suggested that feeling tasks (communication) should be reserved for humans. This research did not account for the potential amplification of biases present in the input used to train AI systems. Clinical research has admittedly found that the analysis and processing of heterogeneous data can be problematic (Dwivedi et al., 2021). There are ethical dimensions to consider regarding data sharing and discrimination. Even though humans do not conduct the analysis and decision-making, the AI algorithm can reflect the pervasive discriminatory attitudes of the engineer or the source data (OpenAI, 2019). Challenges related to data usage and integrity has also been highlighted. As technology matures, these issues need to be resolved to ensure full confidence among clinicians and research stakeholders (OpenAI, 2022).

Trends of Bias in the Usage of Artificial Intelligence

Bias in the Usage of Artificial Intelligence across Different Populations

In one study, Johnson and Williams (2023) assessed GPT-4’s potential to perpetuate racial and gender biases in clinical decision-making. A team of Brigham researchers analyzed GPT-4’s performance in four clinical decision support scenarios: generating clinical vignettes, diagnostic reasoning, clinical plan generation, and subjective patient assessments. The study found that GPT-4 has the potential to perpetuate racial and gender biases in clinical decision-making. The authors suggest that further research is needed to understand the extent of these biases and how they can be mitigated. Clinicians already face considerable pressure to diagnose and treat patients accurately and fairly. Relying on AI tools prone to bias could introduce additional challenges, requiring clinicians to constantly evaluate and potentially override AI suggestions, adding to their workload. This could also create ethical dilemmas when faced with conflicting recommendations.

In my usage, language models have been shown to amplify biases and perpetuate stereotypes in cases against women more than men (Blodgett et al., 2020; Onah et al., 2024). Like earlier GPT and other common language models, both early launch versions of GPT-4 have been found to have the potential to reinforce and reproduce specific biases and worldviews, including harmful stereotypes and demeaning associations for certain marginalized groups (Onah et al., 2024). Model behaviors, such as inappropriate hedging, can also exacerbate stereotyping or demeaning behaviors. In recent decades, there has been a growing trend of incorporating data collection across various aspects of life, facilitating the development and application of new AI methods (California Association for School Psychologists, 2020). This trend is evident across domains with prevalent multi-dimensional data sets and common use of AI methods. Researchers and clinicians from various disciplines have increasingly highlighted a pervasive assumption in relying on these methods (O’Neil, 2016; Prakash et al., 2022). Hence, this could undermine the trust between therapist and client, impede the therapeutic process, and perpetuate harmful stereotypes within the therapeutic setting.

Research on multi-dimensional databases and algorithms has shown they are prone to biases and heuristics, relying on arbitrary classifications, messy data, and numerous concealed uncertainties (Hong, 2020). Similarly, psychotherapy data can also be biased. For instance, cognitive behavioral therapy has shown to be predominantly developed and utilized with White, well-educated, heterosexual individuals (Wong, 2023). Any algorithm based on such historical data risks ignoring large segments of the population, including neuro-diverse individuals, racial and ethnic minorities, culturally diverse groups, LGBTQ+ individuals, and people from diverse socioeconomic backgrounds (Wong, 2023). Additionally, factors such as history, background, lived experiences, and context are crucial in psychotherapy. Consequently, many environmental factors important to mental health treatment outcomes were not considered. This is a significant limitation for all current data sources in psychotherapy practice, research and AI. This emphasizes another important point: data only provides a limited perspective of the real world. The application of machine learning here tends to automate homogeneity, marginalizing the humans whose lives the data represent (Crawford, 2021).

Deductively, it can be said that AI platforms have been shown to have inherent biases and discriminatory factors. AI algorithms, as presently developed, are based on a set of data that represent society’s historical and systemic biases, which ultimately transform into algorithmic biases. Even though the bias is embedded into the algorithmic model with no explicit intention, it is clear that there are various cultural and racial biases in different AI-based platforms (Akgun & Greenhow, 2021). For example, chatbots have been shown to reproduce stereotypical gendered language, such as referring to nurses as “she” and doctors as “he” (Bastiansen et al., 2022; Nag &Yalçın, 2020).

Bias of Artificial Intelligence Among Clinicians and Practitioners

A study by Chekroud et al. (2024) highlights the limitations of clinicians and psychotherapists using an algorithm to predict outcomes for schizophrenia treatment, challenging the assumption of algorithmic infallibility. The literature contains numerous examples of algorithms that harm vulnerable and marginalized groups, even when they perform as intended (Broussard, 2018; Eubanks, 2019). This reveals that a sentencing algorithm exhibited bias against Black individuals, perpetuating historical racial injustices by predicting a higher risk of recidivism for Black individuals and a lower risk for White individuals, resulting in longer and harsher sentences for Black people (Eubanks, 2019).

Further, clinicians’ experiences, emotions, and psychological states are inherently complex and influenced by numerous factors that extend beyond quantitative data (Farahany, 2023). While machine learning and AI algorithms excel at analyzing large datasets and making inferences, they often struggle to capture the nuances, context, and subtleties inherent in human behavior and clinicians’ expertise (Ghandeharioun et al., 2019). By placing trust in AI inferences, there is a risk of neglecting essential clinician experience, interpersonal dynamics, cultural contexts, and other important variables. Any potential oversimplification may lead to misguided interventions or misinterpretations of the clients’ needs. Safeguarding against potential biases within machine learning models is crucial to prevent unintended harm or the exacerbation of existing disparities in healthcare.

Also, human biases often lead us to take the shortest cognitive route irrespective of our expertise or professional qualifications, facilitating irrational investment in algorithmic systems (DeNardis, 2020). This risk is exacerbated by creating increasingly complex data sets that encompass previously unquantified aspects of people’s lives—physical, neurological, psychological, and emotional (Farahany, 2023). Overreliance on algorithmic inferences could inadvertently diminish the essential human element of psychotherapeutic interventions (Renieris, 2023). If we fail to address this overreliance on algorithmic issues, we will develop AI that appears intelligent on the surface but is rife with injustice and inequity underneath, potentially oversimplifying the human complexities (social, cultural, political, historical, and personal) that are central to psychotherapy as a human endeavor. The complexity of new technology might shift the underlying conceptions of autonomy of clinicians, beneficence, non-maleficence, and justice, or might introduce new normative and conceptual distinctions. Causing harm in a traditional setting might mean something different than causing harm in a digital world. For example, the integrity of psychologists and clinicians is one of the core principles in psychotherapy (American Psychological Association, 2017), and forms of deception are justified only under exceptional circumstances. However, is it justified for a chatbot to interact as if it were empathetic? Some literature calls for more in-depth studies, holistic and human-centered approaches, and research focused on the long-term societal and individual impacts of novel technology (Wong, 2023).

Moreover, incorporating AI-based technology into psychotherapeutic practice could have profound philosophical impacts, potentially altering humanity’s collective identity and basic conceptions of knowledge, life, reality, and existence. From the foregoing, it is evident that the field of psychotherapy faces numerous significant challenges. First, the growing prevalence of mental health conditions strains service delivery due to a shortage of trained professionals across different cultures and races experienced in the use of AI systems. This scarcity makes it difficult for clients to access evidence-based treatment options, a situation exacerbated by the recent pandemic (Santomauro et al., 2021). Second, not everyone benefits from psychotherapy treatment with about 50% responding to treatment, and about 30% experiencing are mission of symptoms (Santomauro et al., 2021). Despite significant advancements, diagnostic clarity and prognosis prediction remain elusive.

Recommendations on Bias Projected by Artificial Intelligence

Cultural competence training

Institutions responsible for AI deployment and usage should train clinicians in cultural competence so they can recognize and mitigate biases when using AI tools in therapy (OpenAI, 2022). This can include creating tool adaptation by considering the cultural contexts and nuances that might affect their performance and relevance, thereby facilitating bias. Clinicians, in general, should be trained in identifying demographic or clinical populations at greatest risk of engaging in harmful and potentially fatal behavior and bias, thus using AI appropriately. With these trainings, clinicians can consequently investigate dynamic intra-individual factors associated with an increased risk of harmful behavior for the individuals and population in general (Onah et al., 2024).

Diversifying training data

One of the primary sources of bias in AI models is the data used to train them. To mitigate this bias, datasets should be diversified with the training data including a wide range of cultural, racial, and ethnic backgrounds. This helps the AI system learn from a variety of experiences and perspectives. Further, augmenting existing datasets with synthetic data that reflects underrepresented groups to balance the dataset is a key way of mitigating bias. Regular audits to continuously analyze and update the datasets to ensure they remain representative of the target populations are also necessary.

Ensuring explainability and interpretability

Ensuring the explainability (or interpretability) of AI models is a primary consideration for the usage of AI in psychotherapy. This involves clearly explaining the model’s mechanisms and outputs to another human, including any inherent biases. When predicting clinical outcomes, it is crucial to understand definitions, clinical measures used, and temporal considerations for achieving acceptable accuracy (e.g., 85%) for balanced trade-offs (Prasad et al., 2023).

Addressing socioeconomic and demographic biases

Studies have revealed that race and ethnicity, poverty, and living in rural areas are associated with the exacerbation of pediatric mental health issues (Kretzschmar et al., 2019). Mental health professionals using AI-assisted technology should vet the technology’s creator to ensure steps have been taken to protect against harmful biases in the training data sets and algorithms (Rahman, 2023) in their cultures and countries. The most essential variable in producing non-biased AI is the diversity of the team that built it.

Legal and ethical considerations

Researchers and clinicians need to be aware that new laws in the European Union and African Union seek to ensure AI systems used in these regions are safe, transparent, traceable, non-discriminatory, and overseen by people to prevent harmful outcomes (European Parliament, 2023). It is important for psychotherapists, psychologists and clinicians alike to understand the relevance of these laws in the context of AI and psychotherapy practice (Fiske et al., 2020).

Data integration and privacy concerns

Researchers like Chekroud et al. (2024) amalgamate different databases in psychotherapy, a practice currently observed in the field. However, inherent limitations exist regarding data completeness, prompting debates on datafication (Ulberg et al., 2023). While some advocate for a complete ban due to privacy concerns and stereotypes, it is important for innovation potential, with privacy-enhancing technologies offering a secure integration of diverse sources (The Royal Society, 2023).

Synthetic data as a solution

Synthetic data offers benefits like customizability, cost-effectiveness, rapid production, privacy protection, and inclusion of diverse groups. While AI has progressed in drug discovery, its role in psychotherapy is still emerging and not yet suitable for essential school psychological services, such as assessment, therapy, and supervision. These responsibilities lie with school psychologists and licensed educational psychologists who should select tests and provide clinical judgments (California Association of School Psychologists, 2020). AI should not replace supervisors in training. Direct feedback from supervisors is crucial to ensure high-quality training and uphold practice standards (California Association of School Psychologists, 2015).

Scientist–practitioner paradigm

Adhering to the scientist–practitioner paradigm, clinicians should analyze data with AI within the established framework of psychotherapy theory and practice. Formulating questions and hypotheses based on evidentiary standards aids in interpreting results and discerning inherent data quality limitations. Implementing safety features, like out-of-distribution detection in machine learning models, ensures responsible deployment, preventing inappropriate predictions for individual clients (Chen et al., 2020).

Conclusions

It can be seen that AI’s sophisticated methods come with significant computational costs and bias, posing challenges in some cultures. As AI intersects with psychotherapy, it is crucial to acknowledge potential risks and biases through evidence-based practices. These challenges emphasize the need to address treatment access and bias, and to improve treatment effectiveness. Although AI-supported psychotherapy offers benefits in efficiency and access, its costs and variable impacts across cultures must be carefully considered to enhance its validity and reliability. By implementing these strategies, developers and practitioners can work towards minimizing bias in AI-driven psychotherapy tools, thereby enhancing their fairness and effectiveness across different cultural and racial populations.