Chatbots using conversational AI are making it harder to distinguish climate misinformation from real science, prompting climate experts to use similar tools to detect fake information online. However, general-purpose large language models (LLMs), such as Meta’s Llama and OpenAI’s GPT-4, are lagging behind models specifically trained on expert-curated climate data.
To evaluate these models, researchers used a dataset known as CARDS, consisting of paragraphs from climate-skeptic websites and blogs. The paragraphs fell into different categories of misleading claims about climate change. The team fine-tuned OpenAI’s GPT-3.5-turbo3 on the CARDS dataset to create a climate-specific LLM. The performance of this fine-tuned model was compared to other general-purpose LLMs and an openly available RoBERTa model trained on the same dataset.
The fine-tuned GPT model performed the best in classifying misleading claims, scoring 0.84 out of 1.00 on the measure scale. The general-purpose GPT-4 models and the RoBERTa model scored lower, indicating that including expert feedback during training improves the models’ performance. Non-proprietary models tested by Meta and Mistral performed poorly due to computational constraints, highlighting the need for accessible resources for climate organizations to use more powerful models.
Testing the fine-tuned model on 914 paragraphs about climate change on low-credibility websites showed high agreement with categories identified by climate communication experts. However, the model struggled to categorize claims related to the impact of climate change on animals and plants, indicating a potential limitation in the training data. Additionally, the constantly evolving nature of climate misinformation poses a challenge for generic models to keep up with shifts in the information being shared.
The findings suggest that climate organizations need to carefully consider the models they use in chatbots and content moderation tools to effectively combat climate misinformation. By involving relevant experts in the training process and leveraging climate-specific LLMs, organizations can enhance their capabilities in detecting and classifying false or misleading claims about climate change. Governments are also urged to support the development of open-source models and provide resources to address the growing challenge of climate misinformation in the digital space.
