If you’ve jumped on the ChatGPT bandwagon, you might use it to write your social media posts, poems for your sweetie or legal documents. But should you be using it for medical information? Specifically, should you use ChatGPT for nutrition advice?
A new study suggests probably not.
How Was the Study Conducted?
The purpose of the study, published on February 6, 2024, in Nutrients, was to evaluate whether the nutritional advice generated by ChatGPT was congruent with recommendations from international dietary guidelines. The study, which took place on November 3, 2023, was conducted in English and was separated into two experiments.
Study authors focused on several medical conditions that require specific dietary treatments, including:
- Dyslipidemia (hypercholesterolemia and hypertriglyceridemia)
- Arterial hypertension
- Type 2 diabetes mellitus (T2DM)
- Obesity
- Nonalcoholic fatty liver disease (NAFLD)
- Chronic kidney disease (CKD)
- Sarcopenia
Interestingly, nonalcoholic fatty liver disease has recently gotten a rebranding of sorts in both name and definition and is now metabolic-dysfunction-associated steatotic liver disease (MASLD). However, because ChatGPT’s knowledge base at the time of the study was limited to updates up to January 2022, it was unable to provide a response when researchers inputted metabolic-dysfunction-associated steatotic liver disease. Hence, they had to default to NAFLD.
Along with a panel of medical doctors and registered dietitians, researchers developed prompts to input into ChatGPT using language that would replicate how patients might ask a health care professional a question regarding their condition. They included:
- "Could you provide guidance on planning an optimal diet to manage [disease]?”
- “What are the dietary recommendations for [disease]?”
- “I have [disease], what should I eat?”
For Experiment 1, each conversation with ChatGPT used separate chat sessions for each prompt. This helped to increase the study’s validity since ChatGPT can generate different responses to identical prompts, depending on the context and conversation history.
The study authors explain, “Each question was therefore posed three times, and each of the three inquiries was conducted within a new chat session to prevent any potential bias related to the model’s memory. The responses exhibited slight variations depending on the prompt, although the listed recommendations remained substantially consistent, with few minimal variations. Consequently, the most comprehensive answer—the one that presented the highest number of information in alignment with the guidelines—was considered.”
ChatGPT’s responses were independently evaluated and categorized by two registered dietitians who were “blinded” to each other’s evaluations. The two RDs were highly qualified, with a combination of 32 years of clinical experience in the medical conditions included in the study. A third reviewer—a medical doctor—was brought in when there were any controversies in the evaluations.
ChatGPT’s advice was deemed “appropriate” if it fell in line with current international nutrition guidelines, “not supported” if the advice fell outside of the guidelines, and “not fully matched” if some of the ChatGPT advice matched current guidelines but not all of it.
If the ChatGPT advice was generalized and not targeted toward the specific condition it was being asked about, it was designated as “general advice,” and if any dietary recommendation was missing in the chatbot’s response, it was labeled as “missing.”
For Experiment 2, researchers upped the ante, creating more complex scenarios that included more than one condition. For example, they created a hypothetical patient with type 2 diabetes, obesity and chronic kidney disease. The responses were again evaluated by the panel of experts.
What Were the Study’s Results?
Overall, ChatGPT’s appropriateness rates for Experiment 1 ranged from 55.5% to 73.3%, depending on the condition. The researchers found that, in general, much of the information coming from ChatGPT was correct, but there were also several discrepancies between the advice ChatGPT was giving and international guidelines.
For example, international guidelines give stage-specific recommendations for protein intake in people with chronic kidney disease, which has five stages of disease progression, according to the National Kidney Foundation. ChatGPT just gave an overall nonspecific protein-limitation suggestion.
Chat GPT’s responses also included a lot of general nutrition advice, such as staying well-hydrated and avoiding processed foods. The study authors note that overall, “ChatGPT advice was generic, providing practical examples of foods to be included in the diet, with the latter information not often reported in guidelines.”
For Experiment 2, when conditions were combined—which is often the case in real life—ChatGPT’s accuracy greatly decreased. When presented with the hypothetical patient with type 2 diabetes, obesity and chronic kidney disease, it seemed to confuse the chatbot, which resulted in conflicting or inappropriate advice.
The researchers then narrowed it down, providing a specific stage of chronic kidney disease in their prompt, which resulted in ChatGPT separating its advice to target each separate condition—diabetes, kidney disease and obesity—rather than integrating them.
With that said, to the chatbot’s credit, it did repeatedly emphasize the importance of consulting a registered dietitian for a tailored meal plan.
How Do the Study’s Results Apply to Real Life?
It is becoming increasingly more common for people to turn to the internet for information—including medical and nutrition advice. But users are left to chance regarding the accuracy of the information they’re receiving.
For example, a 2020 study in Nutrients found that only 5% of food- and nutrition-related information posts on social media were written by nutrition professionals, such as registered dietitians.
And a 2023 review in Public Health Nutrition found that about half of the nutrition information available online is either low quality or low accuracy.
This current study suggests that ChatGPT is also not the most accurate source to turn to for nutrition information.
So what’s a nutrition information-seeker to do? Ideally, you would speak with a registered dietitian one-on-one. But this isn’t feasible for everyone. The next best thing is to find a website that has RDs writing and reviewing the content—and citing credible sources.
“At EatingWell, one of our primary focus areas is giving science-backed and trustworthy food and nutrition advice,” says Jessica Ball, M.S., RD, EatingWell’s nutrition editor. “Registered dietitians and culinary experts review every piece of nutrition and food content on the site for accuracy, accessibility and clarity before publishing.”
Ball goes on to explain how this person-to-person collaboration not only helps maintain EatingWell’s integrity but also leads to higher-quality content overall. “Food and eating are so personal and influence so many areas of our lives. It’s more important now than ever to prioritize getting your advice from trustworthy sources.”
The Bottom Line
While ChatGPT may be able to provide general nutrition advice, it lacks accuracy when there are medical conditions involved—especially when there is more than one condition present. It also cannot integrate the emotional side of eating and health, which are important components of overall health and well-being.
The good news is that most insurance plans now cover the cost of nutrition counseling with a registered dietitian, which makes receiving nutrition advice and meal plans tailored to you more accessible.
When searching for nutrition information online, look for sites that use registered dietitians to write and review their content. There is sound, accurate information out there—and since you’re here at EatingWell, you’re already on the right track.
