Why So Many Social Science Claims Are False

Authored by Christopher J. Ferguson via RealClearInvestigations,

One of the most urgent questions facing society today is the potential impact of AI on education and labor markets. So it’s no surprise that two scientists attempted to provide an answer, producing a meta-analysis suggesting that incorporating ChatGPT in schooling dramatically improved student outcomes. Social media and online sources promoted the 2025 study as a beacon of bright light.

The problem is that the researchers’ conclusions were later found to be unreliable, and the study was retracted in April – another blow to the public’s faith in science.

Scientists are increasingly chasing answers to big questions in fields from health care to climate change to AI. The academic research industry incentivizes them to produce the biggest possible findings that make for splashy headlines and career promotions. But such discoveries are rare in science, and too often, the result is widespread exaggeration, dodgy methods, and miscommunication.

“The incentive structure of modern science is such that a ‘simplify, then exaggerate’ strategy has become dominant, even if only tacitly,” according to a recent academic review. “To get published in leading journals, to be awarded grants and to be hired as a postdoc or faculty member, a system-wide bias for novelty, exaggeration and storytelling has emerged.”

As RealClearInvestigations has previously reported, an alarming amount of poor-quality academic and scientific research has been discovered in recent years. Hindawi, formerly a part of major science publisher Wiley, had to retract thousands of science articles, many of them alleged to have been produced by paper mills that generate reams of reports of studies that never existed.

While cases of outright fraud have received wide attention, a more subtle infection has spread within the research community: Research results that are inconsistent, based on poor methodologies, or are of limited practical impact are too often transformed into big, broad, but unreliable claims that, over time, reduce public confidence in science.

Several recent high-profile retractions of scientific papers highlight how broad scientific claims can be built upon questionable data. A recent paper asserted that banning sex work increases rapes. The sex work paper got significant attention on social media before skeptical scholars pointed out they couldn’t replicate the findings. In another major case, economists estimated that climate change would have a devastating economic toll on the world in a paper that was widely covered and influential in banking. However, it was ultimately retracted after other scholars alleged major errors.

In each case, science sleuths found that the dramatic claims did not reflect reality. While the number of retractions has soared over the past two decades to several thousand a year, according to the watchdog group Retraction Watch, experts believe only a small fraction of questionable work gets caught.

Flip a Coin

Lee Jussim, distinguished professor of psychology at Rutgers University and frequent critic of social science, has estimated that the vast majority of psychology research claims are false. He also noted that the ability of psychology to replicate most findings is no better than a coin toss.

At the core of the crisis is a broken system that pressures researchers to produce more and more papers in high-profile journals that favor bold rather than incremental (or null) findings. The incentives include:

• Academic reward systems that grant promotions, tenure, and grant funding to researchers based on the number of publications and bold claims they produce rather than thoughtful scholarship
• Journals that often attract subscription dollars and news attention for flashy findings
• News media that win subscriptions and clicks with dramatic, often surprising claims about human behavior
• Politicians who cherry-pick alarmist science claims to promote their favorite policies while overlooking contradictory evidence

“The growing competition and ‘publish or perish’ culture in academia might conflict with the objectivity and integrity of research, because it forces scientists to produce ‘publishable’ results at all costs,” Daniele Fanelli, a meta-scientist at Heriot-Watt University and the London School of Economics, noted in a recent paper.

The system set up to tamp down sensationalism and flawed findings – peer review – is so overwhelmed by the flood of research papers in recent years that it can’t begin to pull all the weeds from the garden of science. Peer reviewers are busy scholars who are mostly unpaid and stretched too thin, even with the help of emerging AI tools that have their own problems with accuracy. One recent paper estimated that, in any given year, peer reviewers collectively work the equivalent of 15,000 years on scientific papers, which, if actually paid, would cost about $2.5 billion for U.S., Chinese, and U.K. reviewers alone.

Even when errors are detected, correcting and retracting papers is a contentious process that often takes years. Publishers and editors fear reputational damage and lawsuits from researchers, providing few incentives for correcting the record even if fraud is suspected.

Numbers Game

A common type of research exaggeration stems from the promotion of “statistically significant” findings. The seriousness of the phrase may lead the public to interpret the findings as reliable and practically useful. But often that’s not the case. That’s because most social science and medical findings are very weak, so weak they might be produced by methodological flaws (or crud) in those studies. Scientists are nonetheless incentivized to highlight the supposed importance of statistically significant studies in order to get attention, even if they don’t warrant it.

As an analogy, consider the difference between a butterfly and an anvil landing on one’s head. Both exert measurable weight on the head and, as such, both produce “statistically significant” results. But the magnitude of impact, the effect size, differs enormously. Most people would be far more concerned to avoid an anvil landing on their head than a butterfly. That’s why “statistically significant” tells us nothing about the practical importance of a finding.

Even a finding with a standard effect size may not be reliable. For instance, social psychologist Daryl Bem published a report in one of the American Psychological Association’s leading journals suggesting that ordinary people had extrasensory perception, or the ability to predict the future. Such an implausible finding raised considerable controversy at the time.

The problem was that Bem used weak research methods common throughout the social sciences to publish impossible results. Long a defender of ESP, Bem had in an earlier paper explicitly noted that the effects of ESP appeared similar to findings in other areas of psychology, saying, “The mean effect size is quite respectable in comparison with other controversial research areas of human performance.” Bem did not respond to requests for comment.

Professor Fanelli said that poor training and unrealistic expectations help explain why social scientists miscommunicate data. “I think that part of a useful cultural change would be towards a greater understanding and acceptance of the limitations of social science,” Fanelli said in an email. “Instead, a lot of it seems to have in some ways reinforced what was deemed the ‘cult of statistical significance,’ a mechanical and simplistic use of [statistical significance] as a paragon of validity.”

Behind this presentation of hyped results is an academic publishing industry of five big companies with handsome profit margins. The Big Five, as they are known (Elsevier, Sage, Springer Nature, Taylor & Francis, Wiley), along with research grant funders, academic tenure evaluations, and the media incentivize scholars to stretch for the dramatic and oversell weak results. In science, a negative finding that shows no effect is as important as a positive one. But few negative findings get published.

Policymakers tend to make things worse. Catherine Knibbs, a psychotherapist and founder of Children and Tech, had been involved in some of the conversations around the controversial U.K. push for social media bans for youth under 16. She suggests policymakers often approach science with set conclusions, not open questions, and appear incurious about data that doesn’t fit their agendas.

“To be honest I feel science is complex and whilst I love that about it, I [feel] quite alone in this,” Knibbs said in an email. “People are often skimming the stats and looking for [evidence that confirms their beliefs]. Not many people have time to slow down and use their slower cognizant processes. It’s why we have the chicken little sky is falling down shit.”

Pushing for Reform

The good news is that a growing number of researchers are pushing for reforms that could improve the integrity of scientific findings. One promising sign is the growth of the “open science movement,” which encourages the preregistration of studies. This reduces the ability of researchers to mangle their data analysis to produce preferred results. And the movement’s promotion of greater data transparency allows for more confirmation of findings.

Brian Nosek, a professor of psychology at the University of Virginia who is leading the open science movement, struck a hopeful tone. “My most optimistic take is that we have established a stronger culture of self-scrutiny and continuous improvement,” he said. “I don’t know how successful the practice changes have been, but I believe that experimenting with new research methods and committing to evaluating what works and what doesn’t will lead us in positive directions.”

Some scholars are breaking away from the Big Five academic publishers by beginning their own nonprofit scientific journals. In some cases, this has been the result of exorbitant fees that journals charge authors for making papers public. A new federal legal case likewise accuses the Big Five of defrauding the government through unreasonable publishing charges, often paid for by government grants. Aside from the financial costs taken on by scholars, independent journals also potentially free scholars from a heavy focus on positive findings or publication bias.

A small number of research institutions are also trying to directly address some of the problematic incentives. In awarding promotions and tenure, they are putting less emphasis on the number of papers that researchers produce and the prestige of the journals where they are published. Instead, the quality of the paper is what counts.

Professor Ric Ferraro, editor of Current Psychology, emphasized that not all scholars or journals are bad actors. Some try to do the right thing and produce solid research. But breaking this cycle of perverse incentives will be difficult, as too many benefit from it, from scholars and academic journals to news media and politicians.

Until the incentives for research change more broadly, the public will continue to be fed a steady diet of bold claims about human behavior built on an unsteady foundation.

This article is part of an RCI series on The Normalization of Scientific Fraud. Read the previous articles here.