More on New England Journal of Medicine fake formaldehyde scandal

On 20 October, I received an enigmatic reply (above) from the Editor-in-Chief of the New England Journal of Medicine. This was to my letter from April complaining about the publication of a flawed study on e-cigarettes and formaldehyde in the NEJM. His note didn’t say much, but it was copied to around 40 others, so I thought I ought to reply. It is an opportunity to write explaining some of the fallout.

You can refresh your memory of this sorry episode here and here.

Here is my response… 

The text of the letter as sent is set out below.  It raises six new developments, which I summarised in the covering email.

Summary of issues raised

1. The publication of our full complaint in the journal Addiction, together with a more extensive exchange of correspondence with the authors than was possible in the New England Journal itself. I think you will agree that the authors fail to address criticisms about the fundamental flaw in their work.
2. A communication from a member of the public explaining how the letter led her son to return to smoking. It is important that we do not lose sight of the human cost of badly conducted and poorly reported research.
3. The efforts of the government public health agency, Public Health England, to address the misinformation created by the paper as part of its wider initiative to review scientific evidence related to e-cigarettes. It should be a source of some concern that official agencies are having to correct and dismiss the findings and hyperbole generated by the study.
4. The use of the highly misleading findings by activists to lobby for a total prohibition of e-cigarettes in Malaysia. It should be a matter of some concern that governments are being misled by this paper, and that it is being used to justify banning products substantially less harmful than cigarettes, while cigarettes would remain on widespread open sale.
5. The misallocation of $3.5 million in American public money to the Portland University group behind this paper to continue this programme. It seems likely that publication in the New England Journal, and the related publicity, made them appear credible and helped them to secure this grant. However, that apparent success is based on flawed research and should have been a cause for the funder to reconsider rather than to proceed.
6. Research on formaldehyde emissions from e-cigarettes by the tobacco company Philip Morris International, which appears to be more credible and carefully executed than the study published in the New England Journal. I hope you recognise the reputational risks for any journal that is found to be less trustworthy than a tobacco company.

I suggest that the NEJM revisits the case for retraction with some care, publishes or commissions a replication study, and challenges the authors more robustly on the flaws in their work.

Letter to Editor of New England Journal of Medicine

[PDF of letter]

Jeffrey M. Drazen, M.D.
Editor-in-Chief
New England Journal of Medicine
Distinguished Parker B. Francis  Professor of Medicine
Harvard Medical School

16 November 2015

Dear Dr Drazen,

Thank you for your reply of 20 October to my email of 21 April, suggesting that I contact Dr Farsalinos. I am grateful for your response. 

I am of course in contact with Dr Farsalinos. Both he and I remain firmly of the view, along with many others, that the Jensen et al paper on ‘Hidden Formaldehyde in E-cigarette Aerosol‘ is fundamentally flawed, compromises the integrity of the academic record and continues to cause damage to the public, the policy process and the reputation of the New England Journal of Medicine. It should not have been published in the first place and it ought to be retracted now. 

If I may, I would like to update you with six new developments since I last wrote, finishing with some suggestions on how to proceed.

1. Case for retraction laid out in the specialist journal Addiction

On 9th September 2015, our complaint about this paper was made public and published in the journal Addiction, along with a letter summarising the case for retraction, a reply from the original researchers, and a response to their reply from Dr Farsalinos and me. This exchange is now on the record and at the links below as published in Addiction, October 2015 online version.

1. Letter from Bates/Farsalinos to Addiction outlining the basis of the case for the retraction from NEJM. Research letter on e-cigarette cancer risk was so misleading it should be retracted
2. Reply from some of the original authors of the paper published in NEJM (Pankow et al): Formaldehyde from e-cigarettes—it’s not as simple as some suggest.  You may judge for yourself, but this reply is woefully inadequate and fails to address or acknowledge the weakness in their method.
3. Further response to Pankow et al, by Bates/Farsalinos reiterating the central flaw in their work. E-cigarettes need to be tested for safety under realistic condition

I have written up the case so far in full: Flawed e-cigarette formaldehyde paper should be retracted – formal complaint and supporting letter published

The case for retraction remains undiminished by the responses from the authors but has been strengthened by further studies, confirming the high likelihood of the measurements were made in dry-puff conditions (see 3 below) and should not, therefore, have been used in a calculation of human cancer risk. The experiment suffers from a glaring methodological error that the authors did not, and apparently still do not, understand. For a calculation of the risk of cancer, a condition of the human body, the estimation of exposure has to be realistic for human users. May I suggest that you revisit the case for a retraction in the light of these developments?

2. Misleading information has adverse impact on smoking – a letter from a member of the public

It is important that authors and journal editors are aware of the human consequences of misinformation on smoking-related issues. It is hard to survey this in detail, but a single anecdote is telling. Please see this email sent to Dr Farsalinos from a member of the public, and presented on his website: Portland university formaldehyde scandal update: an email from a consumer

It is important to be aware that actual bodily harm can arise from the way studies are conducted and reported, and that authors and editors should be responsible and accountable if they have been negligent. Given the health risks, there is a particular imperative to retract studies with serious flaws that would tend to promote adverse changes in smoking behaviour.

3. Public Health England’s efforts to clear up confusion created by the Hidden Formaldehyde letter

The main public health agency in England, Public Health England (the CDC equivalent), recently commissioned an evidence review of e-cigarette science. May I draw your attention to the discussion on page 77 of the main review:

In January 2015, a similar report was published as a research letter to the New England Journal of Medicine (NEJM) [133]. In this study, negligible levels of formaldehyde were released at lower EC [electronic cigarette] settings, but when a third generation EC (EC with variable power settings) was set to the maximum power and the apparatus was set to take puffs lasting 3–4 seconds, this generated levels of formaldehyde that, if inhaled in this way throughout the day, would exceed formaldehyde levels in cigarette smoke between five and 15 times. 

The EC was puffed by the puffing machine at a higher power and longer puff duration than vapers normally use. It is therefore possible that the e-liquid was overheated to the extent that it was releasing novel thermal degradation chemicals. Such overheating can happen during vaping when the e-liquid level is low or the power too high for a given EC coil or puff duration. Vapers call this phenomenon ‘dry puff’ and it is instantly detected due to a distinctive harsh and acrid taste (it is detected by vapers, but not by puffing machines) [139]. This poses no danger to either experienced or novice vapers, because dry puffs are aversive and are avoided rather than inhaled. 

A study has just been published testing the hypothesis that the NEJM report used dry puffs [140]. An equivalent EC product was set to the same or normal settings and used by seven vapers. The vapers found it usable at normal settings, but all received dry puffs and could not use the device at the settings used in the NEJM report [133]. The product was then machine tested. At the dry puff setting, formaldehyde was released at levels reported in the NEJM letter and the Japanese press release. At normal settings, there was no or negligible formaldehyde release. 

The review concluded:

There is no indication that EC users are exposed to dangerous levels of aldehydes.

It should be a matter of some concern, or even embarrassment, to the editorial board at the NEJM that a prominent government public health agency feels compelled to point out the flaws in this study and to discount its headline findings completely.

A major concern of Public Health England was to clear up public misperceptions of e-cigarette risk created by poor science or misreporting of studies, given a large and increasing proportion of adults both in UK and US erroneously believe that there is no difference in risk between smoking and e-cigarette use – thus leading to the adverse behavioural changes discussed above.

It was notable, though unsurprising, that Strongin et al of Portland University joined in criticism of Public Health England.  As ever they did little more than add to the impression that they do not know what they are doing, as I show in my response to their ill-founded criticisms.

• Harm Reduction is Not Well-Served by Bias (Strongin, Peyton, Pankow)
• PHE is obliged to challenge false risk perceptions created by poor science and sensationalist communication (my reply)

4. Hidden Formaldehyde letter used in lobbying governments for e-cigarette prohibition

The authority and reputation of the name of the New England Journal of Medicine have been used to lend credence to this study in a campaign to have e-cigarettes banned completely in Malaysia. An op-ed written in the Malay Mail Online used the misleading finding of the study to make its case.  E-cigarettes should be banned — Mary Assunta (4 October 2015).  In this op-ed, Ms Assunta argues her case for prohibiting the much safer product while leaving cigarettes on the market with reference to the Jensen et al Hidden Formaldehyde paper:

in recent times more research on risks of using e-cigarettes have emerged:
* E-cigarette vapour can contain cancer-causing formaldehyde at levels up to 15 times higher than regular cigarettes. The finding was published this year in New England Journal of Medicine.

While we are used to activists misrepresenting published academic papers, regrettably, in this case, she is accurately citing completely misleading claims as originally published in the NEJM.

5. Publication in NEJM supports misallocation of American public money – a $3.5m grant awarded to the authors 

In September, the Portland University research team that conducted this flawed study was awarded a $3.5 million grant by NIH: Portland State nets $3.5 million NIH grant to help clear the air on hazards of e-cigarettes. It is clear from the press release that the publication of their research in a prestigious journal generating considerable publicity had played an important role in their application.

The team’s preliminary research underscored the need for more information. In a study published earlier this year in the “New England Journal of Medicine,” they revealed that vaping devices can provide a previously unknown reaction pathway to deliver high levels of formaldehyde, a known carcinogen.

Given the naivety of the errors made in this study and the extent of the unjustified publicity created, it is of concern that this team will now be funded to provide advice on regulation to the FDA. This problem can be traced back to a failure of quality control at the NEJM in publishing the paper and to the subsequent failure to retract it when its failings were pointed out.  On the strength of this study, they should not have received any NIH funding.

6. Are tobacco companies becoming more reliable than the New England Journal of Medicine?

The tobacco company Phillip Morris International has recently published research undertaken using validated analytical methods in accredited laboratories. The research was careful to avoid dry-puff scenarios and measured 21 different products under two standardised puffing regimes (Health Canada and CORESTA).  The following charts summarise the results – showing all products tested under realistic conditions having formaldehyde emissions below 10% of cigarettes, and most far below that. They emphasise that their methodology excludes dry puff conditions.

As you can see, the e-cigarettes produce less than one-tenth of the formaldehyde emissions from cigarettes, not 5-15 times greater, when care is taken to avoid dry-puff conditions.

It is a reason for some concern, and even despair, that we should now find a multinational tobacco company to be a more reliable source of scientific authority in this field than a manuscript published in the New England Journal of Medicine. I hope you recognise the reputational risks for any journal that is found to be less trustworthy than a tobacco company.

What should happen now?

• In the light of the developments described above, the NEJM editorial board should revisit the complaint and take the case for retracting this study seriously;
• The NEJM should publish or commission a replication study, with the same equipment and settings, but including a test for dry-puff conditions;
• The NEJM should press the authors to address the complaints about the glaring flaw in their methodology. The editorial board should ascertain what steps the authors took to avoid unrealistic measurements in dry-puff conditions that humans would never experience and, therefore, should not be incorporated into cancer risk calculations;
• If it is shown that the measurements were made in dry-puff conditions or in the absence of a satisfactory response to the concerns raised, the Jensen et al letter should be retracted.

Yours sincerely

Clive Bates
Counterfactual

Note: no competing interests

[letter ends]

Update 24 November 2015: another case of claiming e-cigarettes have higher cancer risk than smoking and other arguments spawned by this paper. This time on CapeCod.com.

A look at the dangers of inhaling formaldehyde published in The New England Journal of Medicine earlier this year included two studies that showed that the risk of developing cancer from long-term vaping is five times as high as that of smoking a pack of regular cigarettes a day. The second study suggested the risk was 15 times higher.

“Formaldehyde is present in cigarettes as well, but it’s felt that the exposure of formaldehyde in the e-cigarette compared to a regular cigarette has probably a five times greater risk of toxicity.”

Brutal critique by Mike Siegel.