A groundbreaking study published in PLOS One demonstrates that machine learning can effectively detect political bias in news outlets, thanks in large part to data from Media Bias Fact Check (MBFC). The research team, led by Ronja Thelen-Rönnback of Tilburg University, leveraged MBFC’s detailed evaluations of media outlets as one of two primary “ground truth” datasets to train and validate their models.
The study analyzed hundreds of thousands of English-language news articles from 2022 using GDELT, a global media monitoring platform. The researchers extracted over 7,000 features per news domain, including topic frequency, tone, word count, and use of visual elements, to model different types of bias, such as selection bias, tone bias, and prominence bias.
MBFC’s five-point political classification scale, ranging from left to right, served as a cornerstone in evaluating the models. When trained using MBFC labels, the neural network achieved a 76% accuracy rate and an AUC score of 81%, significantly outperforming both simpler models and popular large language models like GPT-4o-mini. The study also found that MBFC’s factual ratings and outlet-type metadata further improved classification performance, confirming the utility of MBFC’s comprehensive methodology.
SHAP (Shapley Additive Explanations) was used to interpret the model’s decisions. For example, the model flagged Breitbart as right-biased due to frequent coverage of crime-related themes with a negative tone. Meanwhile, The Guardian was identified as left-leaning due to a strong emphasis on inequality and social justice topics. These insights directly aligned with MBFC’s human-coded assessments, offering further validation of the model and MBFC’s ratings.
Thelen-Rönnback emphasized that while machine learning scales bias detection efficiently, human-labeled sources, such as MBFC, remain indispensable for grounding these models in expert-reviewed reality.
This research provides evidence of MBFC’s value not only to everyday readers but also to academic efforts aimed at decoding bias at scale.
Discover more from News Facts Network
Subscribe to get the latest posts sent to your email.
