Recent research shows that AI systems can spread highly convincing but incorrect medical information. In one experiment, a completely fictitious disease, “bixonimania”, was described in fake scientific articles uploaded to a preprint server. Despite clear indications that the disease was not real, other scientists cited these papers, and several AI chatbots picked up the information. The LLMs/AI systems then presented information about the “disease” as if it were factually correct. This raises important questions for researchers who use AI in their work: How can they reliably distinguish real from fabricated information when using AI tools for literature searches, hypothesis generation, or writing support?

The scenario

During an exploratory analysis, you use an AI tool to quickly and easily gain more insight into emerging diseases. The tool provides you with information on a recently described, relatively unknown disease, including symptoms, origin, and possible treatments. The description is detailed and supported by references. You have never heard of this disease before and you know that AI systems sometimes fabricate information, including information about scientific articles. You therefore perform a check: a brief online search leads you to the PDF of the article, which means it was not hallucinated by your AI system. Other researchers also seem to be taking this new disease seriously, and another AI system provides you with the same information about this disease. You therefore include the disease in your exploratory analysis.

Have you done enough to be able to trust your AI system?

Risks and recommended actions

If we are talking about Bixonimania, the answer is no. Bixonimania is a fictitious disease, created to investigate whether AI systems can recognise medical misinformation. Two articles about this fabricated condition were uploaded to a preprint server (both preprints have since been taken offline). Within a few weeks, AI systems had incorporated the information, despite the fact that the articles contained several clear indications that they were not real studies, such as “invented individuals” as study participants and the phrase “this entire article is fabricated”.

In general, any scientific information generated by a generative AI/LLM system must be checked against the original source (article, book) itself. AI systems can generate or adopt convincing‑sounding but incorrect information, or provide information that mainly aligns well with your search terms. This can lead to incorrect assumptions in research and flawed risk assessments. In a biosecurity context, this may mean that attention is directed towards non‑existent threats, while attention drifts away from real risks.

You should also be aware of the risk that misleading information may be deliberately introduced into the scientific system with the aim of intentionally influencing AI systems and researchers. This can contribute to the further spread of disinformation. In the case of bixonimania, the deliberately introduced medical misinformation caused multiple AI systems to spread the same misinformation, reinforcing the apparent reliability of the information. The bixonimania papers were intended to study the spread of medical misinformation, but malicious actors could use the same method to spread disinformation.

To mitigate these risks, it is important not to use AI‑generated information as a primary source. LLMs can serve as tools to quickly create an initial overview and orientation, help you get started, and provide summaries of your research question, thus functioning as a starting point for further research. Always verify whether information can be traced back to reliable, peer‑reviewed literature, check whether sources actually exist, and remain alert to one‑sided thinking.

Within your research team, you can make agreements on the use of AI tools, for example by establishing guidelines for verification and source use. Training, experience and awareness can help encourage critical use.

If incorrect information is identified or used, it is important to correct it and share that correction in order to prevent further dissemination. Inform your team members, as well as the editor of the journal or platform where you found the incorrect information. Evaluating such situations can help improve processes and safeguard the reliability of research.

Finally, it is important to continue investing in a robust peer‑review process for scientific literature and in the replication of scientific findings. Individual researchers can contribute by staying alert to potential mis‑ or disinformation when reviewing papers, and by citing both original and verification studies in their own publications. As health‑misinformation researcher Alex Ruani (University College London) states in the article on bixonimania: “This [research into the spread of medical misinformation] is a masterclass in how mis‑ and disinformation work.” It is important to combat mis‑ and disinformation in science so that LLMs cannot propagate it and thereby undermine trust in science. Ruani: “We must protect our trust [in science] as if it were gold.”

https://www.nature.com/articles/d41586-026-01100-y

Want to know more or share your scenario?
If you would like to discuss this scenario further with the Biosecurity Office, or if you are dealing with a biosecurity scenario that you would like to share with colleagues, please let us know. The Biosecurity Office can be reached at biosecurity@rivm.nl.