AI, Agency, and Healthcare

Artificial intelligence is rapidly transforming from a simple tool into a powerful agent capable of independent decision-making. This “AI agency,” characterized by the ability of software to analyze data, adapt to changing conditions, and make decisions autonomously, has profound implications for healthcare. It raises critical questions about responsibility, accountability, and the very nature of the doctor-patient relationship.

The emergence of agency in healthcare AI presents a paradigm shift demanding careful navigation. The medical community must proactively engage in shaping ethical guidelines and practical protocols to ensure this evolving technology enhances, not replaces, the human element of medicine. In this article we will discuss benefits and risk of agency embedded within the machines that continue to subtlety harness increasing power and ubiquity in our lives and practice.

Defining Agency in AI

The concept of “agency” in artificial intelligence, while increasingly relevant, remains complex and multifaceted. It signifies a shift from AI as a passive tool to an active entity capable of independent decision-making and goal-oriented behavior. Drawing on the work of leading AI researcher Stuart Russell, agency can be defined as “the capacity to act independently and make choices” (Russell, 2019). This implies a level of autonomy and self-direction that goes beyond basic algorithms with simple rule-following or task execution.

Philosopher Daniel Dennett further elaborates on this, suggesting that agency involves “the ability to control one’s own actions and mental states” (Dennett, 1987). In the context of AI, this translates to systems that can not only process information but also choose how to act upon it, potentially adapting their behavior based on experience and changing circumstances. This aligns with Judea Pearl’s emphasis on causal reasoning in AI, where agentic systems can understand and manipulate cause-and-effect relationships to achieve desired outcomes (Pearl, 2009).

It’s crucial to distinguish between AI agency and mere automation. While automation involves the execution of pre-programmed tasks, agency implies a higher level of autonomy and decision-making capability. For instance, an automated insulin pump delivers insulin based on predefined parameters, whereas an agentic AI system could analyze real-time patient data, predict glucose fluctuations, and adjust insulin delivery accordingly, potentially even considering factors like diet and exercise.

Regarding the spectrum of AI and agency functionality within a machine, we could consider the first level being simply a reactive machine which could make decisions based on immediate inputs but lacked memory or learning capabilities. In the clinical setting of sepsis, a reactive AI system could be designed to continuously monitor a patients’ vital signs (heart rate, temperature, respiration rate, etc.) and lab results.   Acting on a continuing flow of data from a patient, a machine could apply a rule-based analysis, and once a predefined criteria is met, act by alerting clinicians, enabling rapid intervention.

A higher level of agency would be where that same machine monitors the data, applies the rule-based analysis and, beyond alerting a clinician, applies a AI-based clinical intervention, such as immediate infusion therapy, based on pre-established algorithms.

Automatic implanted cardiac defibrillators, (AICDs) arguably carry a reasonable degree of agency as they continuously analyze heart rhythms and “decide” when to autonomously apply a behavior (defibrillation) to reach a pre-determined goal (prevent sudden cardiac death) based on pre-established criteria and decision-making capacity.

Higher levels of autonomy, and therefore agency, involving so-called “theory of mind” capacity, (Carruthers, 1996), where AI driven computers could directly communicate with patients to analyze, judge, and infer their needs is not yet available.  Many computer scientists, however, see such functionality occurring in the not-too-distant future. Those same scientists foresee a time where the ultimate level of agency, computer self-awareness becomes a reality with unknown social consequences. (Russell, 2019).

AI Agency in Healthcare: Current Applications and Emerging Trends

Artificial intelligence is transforming every aspect of society. In healthcare we will see movement beyond simple automation towards greater agency in diagnostics, treatment optimization, and even surgical procedures. This section explores current applications and emerging trends in AI agency, highlighting its impact on accuracy, efficiency, and patient outcomes.

Diagnostic Applications

AI agency is revolutionizing diagnostics, enabling faster and more accurate analysis of medical images, prediction of disease risk, and personalization of treatment plans.

• • Image Analysis: AI algorithms can analyze medical images like X-rays, MRIs, and CT scans with remarkable accuracy, often exceeding human capabilities. Google’s DeepMind developed an AI system that can detect over 50 eye diseases from retinal scans with accuracy comparable to expert clinicians (De Fauw et al., 2018). This demonstrates a level of agency where the AI independently analyzes complex visual data and makes diagnostic assessments.

• • Risk Prediction: AI algorithms can analyze patient data, including medical history, genetic information, and lifestyle factors, to predict the risk of developing various diseases. This allows for early intervention and preventive measures. A study published in Nature Medicine demonstrated an AI system that could predict the onset of acute kidney injury up to 48 hours in advance (Tomašev et al., 2019). This predictive capability showcases AI’s agency in proactively identifying potential health risks.

• • Personalized Medicine: AI can analyze individual patient characteristics to tailor treatment plans and predict drug response. This is particularly impactful in oncology, where AI can help determine the most effective chemotherapy regimen based on a patient’s tumor profile (Ekins et al., 2019).

Treatment Optimization

AI agency plays a crucial role in optimizing treatment strategies, personalizing drug dosages, and monitoring patient response.

• • Personalized Treatment Plans: AI can analyze patient data to create personalized treatment plans, considering individual factors like age, genetics, and lifestyle. This is particularly valuable in chronic diseases like diabetes, where AI can help optimize insulin dosages and manage blood glucose levels (Bergenstal et al., 2019).

• • Drug Dosage Optimization: AI algorithms can analyze patient data and drug interactions to optimize dosages, minimizing side effects and maximizing efficacy. This is crucial in areas like cardiology, where precise drug dosages are essential for managing conditions like heart failure (Nielsen et al., 2020).

• • Patient Monitoring: AI-powered wearable devices and remote monitoring systems can continuously track patient data, alerting clinicians to potential problems and enabling timely interventions. This enhances patient safety and reduces hospital readmissions.

Surgical Robotics and Automation

AI is increasingly integrated into surgical robots, enhancing precision, minimizing invasiveness, and improving patient outcomes.

• • Enhanced Precision: AI-powered surgical robots can perform complex procedures with greater precision and dexterity than human surgeons, particularly in minimally invasive surgeries. This leads to reduced trauma, faster recovery times, and fewer complications.

• • Minimally Invasive Procedures: AI enables the development of smaller, more flexible surgical robots that can access hard-to-reach areas of the body, reducing the need for large incisions. This is transforming fields like neurosurgery and cardiology, where minimally invasive procedures are crucial.

• • Improved Patient Outcomes: Studies have shown that AI-assisted robotic surgery can lead to reduced blood loss, shorter hospital stays, and lower complication rates compared to traditional surgery (Yip & Ng, 2019).

The Ethical Landscape: Navigating the Risks of AI Agency in Healthcare

The increasing agency of AI in healthcare presents a host of ethical challenges that demand careful consideration. As AI systems take on more autonomous roles in diagnosis, treatment, and patient care, we must address concerns related to bias, accountability, transparency, and the potential impact on the doctor-patient relationship.

Bias and Fairness

AI algorithms are susceptible to bias, potentially perpetuating or even amplifying existing healthcare disparities. These biases can stem from various sources:

• • Biased Data: If the data used to train AI algorithms reflects existing biases in healthcare, the resulting AI system will likely inherit those biases. For example, if a diagnostic AI is trained on data that predominantly includes images from white patients, it may be less accurate in diagnosing conditions in patients with different skin tones (Buolamwini & Gebru, 2018).

• • Algorithmic Design: The design of the algorithm itself can introduce bias. For instance, if an algorithm for allocating scarce medical resources prioritizes patients based on factors like income or zip code, it could unfairly disadvantage marginalized communities.

• • Human Biases: Even with unbiased data and algorithms, human biases can influence how AI systems are used and interpreted. Clinicians might be more likely to trust AI recommendations for certain patient groups, leading to unequal treatment.

Addressing bias in AI requires a multi-pronged approach:

• • Diverse and Representative Data: Ensuring that AI systems are trained on diverse and representative datasets is crucial for mitigating bias. This includes data that reflects a wide range of demographics, socioeconomic backgrounds, and health conditions.

• • Fairness-Aware Algorithms: Developing algorithms that are explicitly designed to be fair and unbiased is essential. This involves incorporating fairness metrics into the algorithm’s objective function and testing for bias throughout the development process.

• • Human Oversight and Accountability: Human oversight is crucial to identify and mitigate bias in AI systems. This includes regular audits of AI performance, mechanisms for challenging AI decisions, and clear lines of accountability for addressing biased outcomes.

Accountability and Responsibility

Assigning responsibility for decisions made by AI agents is a complex challenge. When AI systems make errors or contribute to adverse outcomes, who is held accountable?

• • The “Black Box” Problem: Many AI algorithms, particularly deep learning models, are opaque and difficult to interpret. This “black box” nature makes it challenging to understand how an AI arrived at a particular decision, hindering accountability.

• • Distributed Responsibility: The development and deployment of AI systems often involve multiple stakeholders, including developers, clinicians, and healthcare organizations. This distributed responsibility can make it difficult to pinpoint where accountability lies.

• • Legal and Ethical Frameworks: Existing legal and ethical frameworks are often ill-equipped to address the unique challenges posed by AI agency. Determining liability in cases where AI contributes to harm requires careful consideration of the roles and responsibilities of all involved parties.

Addressing accountability in AI requires:

• • Explainable AI: Developing AI systems that are transparent and explainable is crucial for understanding their decision-making processes and assigning responsibility. This involves using techniques like interpretable machine learning and providing clear explanations for AI outputs.

• • Clear Lines of Responsibility: Establishing clear lines of responsibility for the development, deployment, and use of AI systems is essential. This includes defining the roles and responsibilities of developers, clinicians, and healthcare organizations in ensuring AI safety and accountability.

• • Legal and Regulatory Frameworks: Updating legal and regulatory frameworks to address the unique challenges of AI agency is crucial. This includes clarifying liability rules, establishing standards for AI safety and transparency, and creating mechanisms for redress in cases of AI-related harm.

Transparency and Explainability

Transparency and explainability are essential for building trust in AI systems and ensuring their responsible use in healthcare.

• • Understanding AI Decisions: Clinicians and patients need to understand how AI systems arrive at their decisions, particularly when those decisions have significant consequences for health and well-being. This allows for informed decision-making and enables clinicians to challenge AI recommendations when necessary.

• • Building Trust: Transparency and explainability are crucial for building trust in AI systems. When patients understand how AI is being used in their care, they are more likely to accept and trust its recommendations.

• • Identifying and Mitigating Bias: Explainable AI can help identify and mitigate bias in algorithms by revealing the factors that influence AI decisions. This enables developers and clinicians to address biases and ensure fairness in AI applications.

Promoting transparency and explainability requires:

• • Interpretable Machine Learning: Using interpretable machine learning techniques that provide insights into the decision-making process of AI algorithms.

• • Explainable AI Tools: Developing tools that can generate clear and understandable explanations for AI outputs, tailored to the needs of different stakeholders (clinicians, patients, regulators).

• • Education and Training: Educating clinicians and patients about AI, its capabilities, and its limitations is crucial for fostering understanding and trust.

The Doctor-Patient Relationship

The increasing agency of AI has the potential to significantly impact the doctor-patient relationship.

• • Shifting Roles: As AI takes on more autonomous roles in diagnosis and treatment, the traditional roles of doctors and patients may shift. Doctors may become more like “orchestrators” of care, overseeing AI recommendations and providing human connection and empathy.

• • Trust and Autonomy: The use of AI in healthcare raises questions about trust and autonomy. Patients may be hesitant to trust AI systems, particularly if they don’t understand how they work. Clinicians may also struggle with balancing their own expertise with AI recommendations.

• • The Human Element: While AI can enhance efficiency and accuracy, it cannot replace the human element in healthcare. Empathy, compassion, and the ability to build trust are essential qualities that remain uniquely human.

Preserving the doctor-patient relationship in the age of AI requires:

• • Human-Centered AI: Designing AI systems that complement and enhance human capabilities, rather than replacing them.

• • Shared Decision-Making: Involving patients in the decision-making process, ensuring they understand the role of AI in their care and have a say in how it is used.

• • Emphasizing Human Connection: Maintaining a focus on the human element in healthcare, emphasizing empathy, compassion, and the importance of the doctor-patient relationship.

Recommendations and Future Directions

The integration of AI agency into healthcare presents both immense opportunities and significant challenges. To ensure these technologies are harnessed responsibly and ethically, we must proactively shape their development and implementation. This requires a multifaceted approach encompassing guidelines for responsible development, robust regulatory frameworks, comprehensive education and training, and a commitment to human-in-the-loop systems.

Guidelines for Responsible Development

The development and deployment of AI agents in healthcare must adhere to strict ethical guidelines, prioritizing transparency, fairness, accountability, and human oversight.

• • Transparency: AI algorithms should be transparent and explainable, allowing clinicians and patients to understand how decisions are being made. This includes providing clear explanations for AI outputs and making the underlying logic of the algorithms accessible.

• • Fairness: AI systems must be designed to avoid bias and ensure equitable treatment for all patients, regardless of race, gender, socioeconomic status, or other factors. This requires diverse and representative training data and ongoing monitoring for bias.

• • Accountability: Clear lines of responsibility must be established for the development, deployment, and use of AI systems. This includes mechanisms for addressing errors, adverse events, and unintended consequences.

• • Human Oversight: AI should be used to augment, not replace, human clinicians. Human oversight is essential to ensure that AI systems are used appropriately and ethically, with clinicians retaining ultimate responsibility for patient care.

Regulatory Frameworks

Robust regulatory frameworks are needed to govern the use of AI in healthcare, ensuring patient safety and addressing potential legal and ethical challenges.

• • Safety and Efficacy: Regulations should ensure that AI systems used in healthcare meet rigorous standards for safety and efficacy, like those applied to pharmaceuticals and medical devices. This includes pre-market approval processes, post-market surveillance, and mechanisms for reporting adverse events.

• • Data Privacy and Security: Regulations should protect patient data used in AI development and deployment, ensuring compliance with privacy laws like HIPAA. This includes secure data storage, de-identification techniques, and patient consent for data use.

• • Liability and Accountability: Legal frameworks need to clarify liability and accountability in cases where AI contributes to harm. This includes defining the roles and responsibilities of developers, clinicians, and healthcare organizations.

• • Ethical Considerations: Regulations should address ethical considerations related to AI agency, such as bias, fairness, and the impact on the doctor-patient relationship.

Education and Training

Healthcare professionals need comprehensive education and training on AI agency, its capabilities, limitations, and ethical implications.

• • AI Literacy: Clinicians should have a basic understanding of AI concepts, including machine learning, deep learning, and different levels of AI agency.

• • Ethical Considerations: Medical curricula should integrate AI ethics, addressing issues like bias, fairness, transparency, and accountability.

• • Clinical Applications: Training should include practical applications of AI in specific medical specialties, equipping clinicians with the knowledge and skills to use AI effectively in their practice.

• • Critical Evaluation: Clinicians should be trained to critically evaluate AI recommendations, considering the limitations of the technology and the importance of human oversight.

Human-in-the-Loop Systems

Human-in-the-loop systems, where AI agents collaborate with human clinicians, offer a promising approach to harnessing the strengths of both.

• • Combining Strengths: AI can excel at tasks like data analysis, pattern recognition, and risk prediction, while humans excel at empathy, communication, and complex decision-making. Human-in-the-loop systems leverage these complementary strengths to achieve optimal outcomes.

• • Enhancing Human Capabilities: AI can augment human capabilities, providing clinicians with valuable insights and support for decision-making. This allows clinicians to focus on patient interaction, personalized care, and the human aspects of medicine.

• • Ensuring Ethical Use: Human oversight in human-in-the-loop systems helps ensure that AI is used ethically and responsibly, with clinicians retaining ultimate responsibility for patient care.

Conclusion

The emergence of agency in artificial intelligence presents a paradigm shift in healthcare, promising unprecedented advancements while demanding careful navigation of ethical and practical challenges. As AI systems evolve from passive tools to active agents capable of independent decision-making, the medical community must proactively engage in shaping their development and integration.

This article has explored the multifaceted nature of AI agency, examining its potential benefits and inherent risks. We have delved into the ethical complexities, emphasizing the need for transparency, fairness, accountability, and human oversight. The challenges of bias in AI algorithms, the complexities of assigning responsibility, and the potential impact on the doctor-patient relationship have been critically examined.

Moving forward, a collaborative approach is crucial. By fostering interdisciplinary dialogue between clinicians, AI researchers, ethicists, and policymakers, we can establish robust guidelines for responsible AI development and deployment. Investing in education and training for healthcare professionals will empower them to critically evaluate AI’s capabilities and limitations, ensuring its ethical and effective use.

Ultimately, the successful integration of AI agency in healthcare hinges on a human-centered approach. While AI offers immense potential to enhance efficiency, accuracy, and personalized care, it cannot replace the human element of medicine. Empathy, compassion, and the ability to build trust remain uniquely human qualities essential to the art of healing.

As we venture further into this uncharted territory, let us embrace the transformative potential of AI while safeguarding the core values of humanism in healthcare. By prioritizing patient well-being, fostering trust, and upholding ethical principles, we can ensure that AI serves as a powerful tool for empowerment, not displacement, in the pursuit of a healthier and more equitable future.

References

• • Dennett, D. C. (1987). The intentional stance. MIT press.

• • Pearl, J. (2009). Causality. Cambridge university press.

• • Russell, S. J. (2019). Human compatible: Artificial intelligence and the problem of control. Viking.

• • Bergenstal, R. M., Klonoff, D. C., Garg, S. K., Weinzimer, S. A., Buckingham, B. A., Bailey, T. S., … & Haidar, A. (2019). Threshold-based insulin-pump interruption for reduction of hypoglycemia. New England Journal of Medicine, 380(21), 2036-2043.

• • De Fauw, J., Ledsam, J. R., Romera-Paredes, B., Nikolov, S., Tomasev, N., Blackwell, S., … & Ronneberger, O. (2018). Clinically applicable deep learning for diagnosis and referral in retinal disease. Nature medicine, 24(9), 1342-1350.

• • Carruthers, P., & Smith, P. K. (Eds.). (1996). Theories of theories of mind. Cambridge University Press.

• • Russell, S. (2019). Human Compatible: Artificial Intelligence and the Problem of Control. Viking.

• • Ekins, S., Puhl, A. C., Zorn, K. M., Lane, T. R., Russo, D. P., Klein, J. J., … & Freundlich, J. S. (2019). Exploiting machine learning for end-to-end drug discovery and development. Nature biotechnology, 37(7), 769-776.

• • Nielsen, P. B., Mortensen, R. N., Jensen, A. B., Grande, P., & Torp-Pedersen, C. (2020). Deep learning for prediction of all-cause mortality at discharge in patients with heart failure. European journal of heart failure, 22(1), 121-128.

• • Tomašev, N., Glorot, X., Rae, J. W., Zielinski, M., Askham, H., Saraiva, A., … & Mohamed, S. (2019). A clinically applicable approach to continuous prediction of future acute kidney injury. Nature, 572(7767), 116-119.

• • Yip, M. C., & Ng, C. F. (2019). Artificial intelligence in robotic surgery. Surgical endoscopy, 33(4), 1030-1038.

• • Buolamwini, J., & Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency (pp. 77-91). PMLR.

• • Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5), 206-215.

• • Char, D. S., Shah, N. H., & Magnus, D. (2018). Implementing machine learning in health care—addressing ethical challenges. The New England journal of medicine, 378(11), 981-983.

• • Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016). The ethics of algorithms: Mapping the debate. Big Data & Society, 3(2), 2053951716679679.

• • Topol, E. J. (2019). Deep medicine: How artificial intelligence can make healthcare human again. Basic Books.

• • Vayena, E., Blasimme, A., & Cohen, I. G. (2018). Machine learning in medicine: Addressing ethical challenges. PLoS medicine, 15(11), e1002689.