How to Report When the Science Is Sketchy. (It Often Is.)

Conflating dishonesty in publication with open access is dishonest. True, the really ridiculous zero-peer-review author-pay journals are nonsense. But they are easy to spot. Why do you give a free pass to the traditional journals, which have had many scandalous episodes recently? Or fail to mention the entire problem with "reputed" journals (in some fields) who publish results accompanied by press releases which later turn out to be wrong? Try learning about such ideas as Fair Open Access and actually talk to someone who knows about Open Access (e.g. Peter Suber). An extremely conservative and poorly thought out article, and you may be showing your age a little too much.

I submit that the passage quoted below is incorrect, or woefully incomplete. Traditional academic journals are financed mainly by universities, mainly by their employment of academics. Employed scholars submit their articles (without charge), serve on editorial boards (mostly without pay), and act as anonymous peer reviewers (without charge). Subscriber fees (received largely from university libraries!) generally cover costs such as printing or a salaried managing editor. (Distributors such as JSTOR perform none of the above functions and serve as mediators who extract what the original political economists would have called rent.) So I am disputing this passage and ask for the author's clarification: "It’s expensive to publish a journal. You need an editorial staff and you need a coterie of trusted experts to review papers and advise on publication. The question is: Who pays for it all? For traditional journals, subscribers do. Or you can pay what can be a hefty fee for a single paper."

The problems with journal manipulation in highly politicized areas, such as climate change are well known. The effort to maintain the purity of Climate Change journals, and exclude any submissions which were critical of theses promoted by important authors e.g. Michael Mann was well demonstrated in the Climate Gate emails. Even the Guardian found many problems with the policing of the field when those came to light. https://www.theguardian.com/environment/2010/feb/09/peer-review-block-sc... See other comments for an example of automatic attempts to discredit other views, not by debating the theory and data, but by guilt by association.

There is a better solution: arXiv. Almost all papers in the physical sciences are posted there. It costs almost nothing to run. Publication and access are free. Papers aren't bogged down in a peer review system that is sometimes constructive but sometimes is an anonymous blackball (scientists are human, and stabbing competitors in the back, or simply rejecting ideas that disagree with one's own, is a temptation some cannot resist). How does it work? Posting is limited to scientists who have been vetted by existing members. In some fields an informal review screens out crackpot papers (I've never seen one make it through). There is also an automated plagiarism screen.

Very informative article, thank you. The realm of pseudo science greatly parallels the rise of pseudo-facts, if you will. The problem, as you pointed out, is that publishing truly peer-reviewed journals, by experts in the field, are expensive- like legitimate news media outlets that have editors. Freebies can just go so far. How do you protect the veracity of research, yet maintain an "open source" for those with an interest to delve into the literature, but not the means? On-line newspapers. such as the NY Times I know struggled with the same issue, and still does. However, what media outlets seem to have done, is to offer substantial discounts to many subscribers for a set period of time, or have a "pay-as you can" policy for many. Perhaps journals can follow this same policy. There are many scientific journals that are published by the same publisher. Perhaps they can offer a single subscription that enables researchers, and research minded individuals, to have access to the myriad of journals they put out- getting more reasonable fees from a larger pool of subscribers. Until publishers figure out a better method to attract those of us that require or desire to use their publications, the predatory issue you describe in the "open source" world, will continue, just like the "fake-news" sites. If they cannot come up with a reasonable solution, then we are all doomed! Just like the news media.

The predatory journals are the result of dereliction of duty by universities. Academics could form cross-university organizations in specific fields to accredit journals. But there are at least two problems: --It would be A LOT of trouble. And since phony journals are everybody's problem, in a way they are nobody's problem. It's like when someone collapses in a crowded place and everyone leaves it to someone else to do something. --Many academics rightly regard the publish-or-perish system as sick. They may regard resume padding by phony journals as the best available way of coping with a bad situation.

“They don’t bother with the pesky expense of reviewers and they publish only online, so their costs are minimal.“ I act as a reviewer for reputable journals and they never give the reviewer a fee. Scientists do this out of great interest in the field, intrinsic ethical principles of contributing to the betterment of their field, and a feeling of reciprocation because other scientists have done this for them. We Get satisfaction in looking for holes in an argument or in the methods or conclusions. An honest scientist believes in the principle that scientific studies and writings need to be vetted for accuracy and legitimacy to make sure the studies conclusions are supported by evidence. If not they are scientifically meaningless. The only people that can judge the merits of the work and find such errors in a study are other experts in that speciic area of science. As reviewers, we scrutinize and criticize. In the end, the article is recommended for publication with minor revision, major revision, more experiments OR even rejection. To the author(s) this often means more work, sometimes a tremendous amount of additional work, but scientists inherently believe the process is meaningful because the study and the article will be better and the field advanced. Scientists who look for fake journals on the internet - ones you just pay f areor and without peer review - are cynical of the process, are afraid of criticism, are lazy and often lacking in principles.

Having been a working scientist for the last ~50 years I've dealt with a lot of papers that aren't what they (or their university press officers) purport them to be. It's typically not hard to sort out the nonsense. Here's a couple of rules I use: 1) If it seems unlikely (like especially if it violates the Second Law of Thermodynamics), it's probably not true. Wait for a confirming study from an independent lab to get excited. 2) extraordinary claims demand extraordinary evidence (I didn't make that one up, but I don't remember who said it. 3) If you've never heard of the authors, their institutions, or the magazine they are publishing in, be more wary than usual. 4) Start by browsing the Methods section of the paper. If it is gobbledegook, or obviously deficient, read no further. 5) Look at the first figure and see if it is plotted deceptively, uses bogus statistics, or if the experiment lacks a proper control, if so, read no further. (Lots of Science, Nature, and Cell papers flunk on these criteria). If there are no obvious red flags then go ahead and read the paper-- but still be wary. Don't invest a million dollars in time, effort, and supplies until you or someone you trust has verified the critical findings. And don't ever believe anything the press release says, even about your own work!

Open Access does not make the costs of publishing a journal disappear. What it does, is to shift the cost of publishing from the readers to the authors. This creates a conflict of interest: the more scientific papers get published, the higher the revenues of the publisher. Consequently, journal feel tempted to subordinate the scientific soundness of manuscripts to the economic pressure. This conflict of interests is intrinsic to the ­author-pays model and is the business model of numerous Open Access pseudopublishers ranging from malodorous to fraudulent. How can we ensure that scientific excellence will prevail? Any remedy must uncouple the publication of manuscripts from the economic interests of the publisher. The scientific enterprises can cut out the middlemen and save costs – while ­enforcing their own standards of ethics and rigorousness. The Swiss Medical Weekly has pledged to follow this model. Neither authors nor readers are requested to pay for publishing and/or enjoying the papers of the Swiss Medical Weekly. The only criteria for the acceptance of a manuscript submitted to the Swiss Medical Weekly are its ­scientific soundness, its novelty, and its usefulness to the readership. No other considerations play any role in the editorial decision process, and all monetary issues are removed from the equation. I hope that, with time, all science journals will follow suit. Adriano Aguzzi, MD, PhD, DVM hc, Editor-in-chief, SMW

If I have to hear more about grain brain, statin brain, cancer meds being intended to kill people from supposedly intelligent and educated people... Unfortunately, there is a market for any nonsense . and BTW a bit of dark chocolate not very much will prob not hurt your diet... There is lots of bous research, terrible legal decisions... More people should beomce art historian/ or musicians/ aka dilettantes--- they make less trouble... And PS when will we rein in Wall Street -- more nonsense. Trading should not be allowed more than one day a week IMO... The rest of the time, stock brokers can scrape chewing gum off the floors of the subway and sidewalk.. Mao had a name for it.- re-education.. My mother called it seeing how the other half lives. Stop despising people whose seem menial... Any job well done (cleaning the bathroom just before I enter) is laudable

The best approach to judging a journal is to look at its impact factor (https://jcr.incites.thomsonreuters.com) which scores each journal by the number of times its articles are cited by others. If a journal is below a certain threshold (which depends on the field), ignore it.

It's even more tricky. Even legitimate journals have made errors and been fooled by deliberately submitted gibberish. So while legitimacy is a good guide, the scientific peer review process can get complacent. There is overlap between open-access journals and a blog-oriented medium like Medium. I recently published a series in The Medium because it was the most appropriate and convenient outlet for something slightly "political." I wasn't trying to fool anyone by associating it with serious-sounding science. I just had something I thought important to say, and needed a place to say it. I hope if people run across it, they will judge it on its merits. More fundamentally, beyond the current duality between "Real" and "False Facts," there are gray areas of indigenous knowledge; civic science; interdisciplinary, systems, and non-reductionist thinking; issues of what to do on important issues when the science is "sketchy" in a different way than in this article's title; important values are at play; the science is seen as controlled by financial interests; the increasing evidence of cognitive biases and acceptance of emotions; and Kuhn's pointing out of lags when conventional science can get caught up in existing but increasingly obsolete paradigms. So, yes, buyer-beware, but in a more critical sense. We need a new way to look at these kinds of issues, maintaining the primacy of science, but not stopping there. We can start, as good science suggests, with remembering to question.

I appreciate that you have seen through Google's scam, many professors have not and require students to get their sources from Google's tool. Even supposedly top rate $100 per year subscription journals, like Nature are publishing junk science these days. The peer review process is flawed, especially when the author(s) get to pick member's of the review panel. Once upon a time, at least in Chemistry, you needed to have 3 independent groups verify your work by repeating your process. If they got the same results, you were published. Now, many reviewers recommend publishing because it looks right and the numbers should add up. They quibble over punctuation more than is the process correct and are the numbers correct. Verifying results by replication is rarely done these days. For a class project I had to submit a paper to one of Nature's letters publication. My research was not far enough along to have verified results so the professor said "make" them up and just say they are unverified results. When I received comments from the review process, 3 said the paper was fine and should be published as is. I called the editor and asked not to be published. I also lost all respect for Nature as 5 out of the 8 people in the class had similar papers published as submitted. I have had papers I have written published in respected journals but that was long ago.

Scientific publishing, and referencing, is primarily done for aliving, not because what gets published or quoted is necessarily worth to be published or quoted. In consequence, some scientists will silently argue that 50% of what is published in 'prestigious' journals is good science, others believe it is perhaps 5%, at best. It depends on own preferences, knowledge about the history of specific field of research, personal rigor, epistemological design and methods, whether one prefers mainstream work or new ideas, just to mention a few aspects. The opportunity to publish in legitimate journals may strongly depend on politics, governance, networking, and whether a paper is in agreement with the current mainstream. One task for the keen science journalist might be to identify excellent pieces of work published by scientists without much political power in less prestigious, sometimes perhaps even predatory journals.

This is only a problem in science, not in the Humanities and Social Sciences. Professors at lower tier institutions are effectively excluded from leading journals, because they don't have the funding to do the experiments that their peer reviewers demand from them. Experts reviewing articles for journals like Nature and Science are Nobel Prize material if not Nobel Prize winners, and routinely demand additional data for supporting any good idea they see. That may sound fair to outsiders, but it isn't. Most peer reviewers know full well that demanding another $1 million worth of experiments effectively excludes all but the richest labs from publishing in top journals, no matter of how good their ideas are. And that means that science profs at poor institutions have to go for lower-tier journals, which pretty much excludes them from applying for top jobs. That's why brilliant scientists who can't get a job in a top lab often rather go work for industry than for a middling state school or poor liberal arts college.

The author says: "Many — probably most — academics who publish in these journals know exactly what they are doing. They are padding their résumés, taking advantage of the fact that colleges may not know if a journal is legitimate or not." How does she know this? My colleagues are careful to avoid these kinds of journals but they are not always easy to spot. We have conversations often about which journals are predatory and which are not and it's hard to figure out more often than we would like to admit. We end up in some cases with: 'When in doubt, avoid,' which leaves a bit of room for error, unfortunately. Tenure committees have to decide which journals are worthy to publish in and which won't count so any academic author trying to rely on this kind of publishing won't get very far in academia. It sounds like journalists need to come to the university library and ask for some training so at least you will be as well prepared as anyone can be and avoid doing research in predatory journals as much as possible, too.

"the pesky expense of reviewers" It should be noted that the scientists reviewing papers are typically not paid for this, but do it as part of general community service, taking time away from their regular work. Although the editors at journals are paid. Also, in my field (space physics), authors generally make their work available for free on the arXiv "preprint" server. i.e. very similar to the final journal version. This partly alleviates the problem of providing access for free.

Gina, your solution will NOT work for everyone. You work for the New York Freakin' Times! Researchers will answer your calls. What the heck are ordinary folks supposed to do? NOT SO SIMPLE!

It shoul be noted that, at least in basic science, it is not a standard practice, even for prestigious journals, to compensate scientists for referee work. This, IMHO unfairly, is not a cost in editing a journal and for profit journals provide considerable, disproportionate earnings to their editors, mostly at taxpayers expens (through grant money). This is also a huge problem in scientific publishing, although not as dangerous as predatory journals.

Surely a clearing house for issuing authentication codes could be established to validate vetted journals. Or does such a site already exist? If not, perhaps legitimate journals could pool their resources to establish a way of digitally “stamping” their publications to help us differentiate the real from the fake.

Thanks for a great article. I will no longer suggest my students use Google scholar. It's disappointing, of course, because I thought there was at least ONE place one could find good science. Back to hunting and sticking with the known good ones. Thanks again.

Gina Kolata--Thank you for today's piece: When the Science is Sketchy. When I saw this I had hoped it would be an acknowledgment of past use of fake science cited in the past by you esp in re your use of The Advancement of Sound Science Coalition as source material as discussed in The Merchants of Doubt. Otherwise I am a Tuesday Science Times addict. Thank you for your involvement. David

As a scientist, my friendly advice to Ms. Kolata: stay away from press releases made by interested parties; never trust only one source of information; do not jump on the first published report of something new; and beware of the political bias, from the radial left to the extreme right, of the scientists.

Not too long ago a metastudy of results published in psychology journals found that something approaching 50% could not be reproduced in attempted replications. The problem isn't merely that sketchy journals publish ridiculously poor paper. It lies at the very core of certain 'scientific' disciplines.

Gina, have scientists on hand to call, to ask about bogus journals. We know them, and can steer you in the right direction. Yes, we know about the practice of padding CVs with independently published papers. Important work will make its way to important journals. Click-bait (chocolate is good for whatever ails you) will not.

For biological science, use PubMed which vets the journals it covers.

Publishing scientific papers should not be as expensive as they currently are. Reviewers are never paid for their work, neither are most associate editors (who frequently have full time faculty positions). Since most articles are downloaded over the internet, the cost of paper and type setting should be minimal. Much of the cost either goes to profit for the publisher (for commercial publishers) or to support other activities (non-profit scientific organizations).

The US National Library of Medicine maintains a list of peer-reviewed journals whose abstracts are displayed on PubMed, the computerized index of the scientific literature. https://www.nlm.nih.gov/pubs/factsheets/j_sel_faq.html#a8

Hi Gina. Nice that you highlight open access journals and their often nonsense pseudoscience. But you are not correct that scholarly journals pay reviewers, or even members of their editorial boards. In academia, such responsibilities are often markers of one's academic legitimacy, and these tasks are undertaken gratis. Believe it or not, not everything is about money. Yours Sincerely, Barbara Milrod, M.D.

The infamous case 1998 of Andrew Wakefield whose series published in The Lancet, caused fear about the measles-mumps-rubella (MMR) vaccine causing autism. parents stayed away in droves. Global Armageddon started with Thomas Malthus in the late 1700's. He predicted global starvation in 100 years. In the 1960's the Club of Rome predicted a global freeze. Samuel Arbesman wrote in his book, "The Half-Life of Facts," that “the tendency of an original scientific publication to present results that seem far more compelling than those of later studies.” Such a tendency has been documented in the medical literature over the past decade by John Ioannidis, a researcher at Stanford, in areas as diverse as HIV therapy, angioplasty and stroke treatment. The writer states: "American Institute of Bisexuality (A.I.B.), a deep-pocketed group partly responsible for a surge of academic and scientific research across the country about bisexuality." The fact that this study is paid for by A.I.B. renders whatever conclusions suspect. In an article, “Misconduct, Not Errors, Cause Most Retractions” by Daniel Akst (October 2012) he stated, “misconduct was most prevalent at the most prestigious journals. The researchers point to intense pressure for scientists to publish; institutions and science funders rely heavily on published articles in making decision about who to give a job or grant.”

Sadly, in fields related to global warming and climate change, it's even worse. Since vast amounts of money are expended presenting sometimes plausible-looking "expertise" funding is readily available to subsidize the illegitimate efforts. Then, since any effort to explain this phenomenon can be twisted on its head - "fake news" - with little or no effort, the real journals and real scientists are under skilled PR attack at any and all times. And now we have Congress, the President, and his cabinet carefully filled with insider industry advocates, the "villains" are supposed to be the real scientists. Just yesterday Scott Pruitt tightened the screws on real science: https://www.nytimes.com/2017/10/31/climate/pruitt-epa-science-advisory-b... Sadly, even that "some" is a misrepresentation. Any non-industry-tied scientist is barred. No advice from the real world is OK.