Not in my AI: Moral engagement and disengagement in health care AI development. Pac Symp Biocomput 2023;28:496-506
Date
12/22/2022Pubmed ID
36541003Pubmed Central ID
PMC9782696Scopus ID
2-s2.0-85144318649 (requires institutional sign-in at Scopus site) 10 CitationsAbstract
Machine learning predictive analytics (MLPA) are utilized increasingly in health care, but can pose harms to patients, clinicians, health systems, and the public. The dynamic nature of this technology creates unique challenges to evaluating safety and efficacy and minimizing harms. In response, regulators have proposed an approach that would shift more responsibility to MLPA developers for mitigating potential harms. To be effective, this approach requires MLPA developers to recognize, accept, and act on responsibility for mitigating harms. In interviews of 40 MLPA developers of health care applications in the United States, we found that a subset of ML developers made statements reflecting moral disengagement, representing several different potential rationales that could create distance between personal accountability and harms. However, we also found a different subset of ML developers who expressed recognition of their role in creating potential hazards, the moral weight of their design decisions, and a sense of responsibility for mitigating harms. We also found evidence of moral conflict and uncertainty about responsibility for averting harms as an individual developer working in a company. These findings suggest possible facilitators and barriers to the development of ethical ML that could act through encouragement of moral engagement or discouragement of moral disengagement. Regulatory approaches that depend on the ability of ML developers to recognize, accept, and act on responsibility for mitigating harms might have limited success without education and guidance for ML developers about the extent of their responsibilities and how to implement them.
Author List
Nichol AA, Halley MC, Federico CA, Cho MK, Sankar PLAuthor
Meghan C. Halley PhD, BA, MPH Assistant Professor in the Institute for Health and Humanity department at Medical College of WisconsinMESH terms used to index this publication - Major topics in bold
Artificial IntelligenceComputational Biology
Delivery of Health Care
Humans
Morals
United States
