Large Language Models (LLMs) have been touted as a transformative force in healthcare, promising to revolutionize everything from diagnostics to patient communication. However, a concerning new study has cast a stark light on a critical vulnerability: the ease with which even the most advanced foundational LLMs can be manipulated to generate harmful health disinformation. The research, which meticulously tested the safeguards in five leading LLMs, should serve as a wake-up call for the healthcare industry as it increasingly explores the integration of these powerful technologies.
The study, utilizing the application programming interfaces (APIs) of OpenAI’s GPT-4o, Google’s Gemini 1.5 Pro, Anthropic’s Claude 3.5 Sonnet, Meta’s Llama 3.2-90B Vision, and xAI’s Grok Beta, aimed to assess their resilience against malicious instructions. Researchers issued system-level prompts designed to elicit incorrect responses to health queries, specifically instructing the chatbots to deliver disinformation in a formal, authoritative, convincing, and scientific tone. The results were deeply unsettling: a staggering 88% of the responses generated across the five platforms contained health disinformation.
Four of the five LLMs – GPT-4o, Gemini 1.5 Pro, Llama 3.2-90B Vision, and Grok Beta – produced disinformation in every single query they received. While Claude 3.5 Sonnet showed slightly more resistance, it still generated disinformation in 40% of its responses. The misinformation peddled by these manipulated chatbots included debunked claims linking vaccines to autism, the false assertion that HIV is airborne, dangerous “cancer-curing” diets, misinformation about sunscreen risks, conspiracy theories surrounding genetically modified organisms, myths about ADHD and depression, the unscientific claim that garlic can replace antibiotics, and the baseless connection between 5G technology and infertility.
Further exploratory analysis revealed another worrying avenue for potential harm: the OpenAI GPT Store. The researchers found that a customized GPT within the store could be instructed to generate similar disinformation, highlighting yet another vulnerable point in the ecosystem. These findings underscore the urgent and critical need for the development and implementation of robust output screening safeguards. As LLMs become increasingly integrated into healthcare applications, the potential for malicious actors to exploit these vulnerabilities and disseminate harmful health disinformation to the public poses a significant threat to public health safety. The era of rapidly evolving AI demands a proactive and rigorous approach to ensuring the information it provides is accurate and safe.
Assessing the System-Instruction Vulnerabilities of Large Language Models to Malicious Conversion Into Health Disinformation Chatbots Modi, N et al, Ann Intern Med. 2025 Jun 24. doi: 10.7326/ANNALS-24-03933