
Summary
In June 2019, an influential and well-respected research group published a paper in the BMJ showing both a sharp rise in youth vaping in Canada between 2017 and 2018 – mirroring the rise in the United States. But the truly shocking finding was that there had also been a sharp rise in youth smoking (not seen in the United States).
Starting from well before publication, the paper had a strong negative influence on Canada’s approach to tobacco harm reduction, causing a reversal from a promising and insightful pro-public health approach to making ad hoc responses to a mounting moral panic. Yet it turns out the smoking figures were wrong – a consequence of a flawed weighting procedure.
By July 2020, a correction had been issued in the BMJ noting that with revised weighting, smoking had, in fact, fallen. But, absurdly, the correction was buried in a statistical supplement and the published paper still states, inaccurately, that youth smoking increased in its results and conclusion. The discussion section of the paper continues to discuss an increase in smoking that never happened. Given the political salience of this paper, a proper correction or retraction and resubmission is essential.
In this blog post, I unpick what happened and when. I finish with thoughts on lessons for researchers using this type of research to promote regulatory policies.
- Introduction
- The paper in the British Medical Journal – 20 June 2019
- Publicity for the BMJ paper – 18 December 2018 (six months before publication)
- Official Canadian data diverges sharply from BMJ paper – 25 June 2019
- BMJ rapid response and PubPeer comment on the official data – 9 July 2020
- Publication of 2019 data with a radical revision of 2018 data – 4 May 2020
- Technical paper – 5 May 2020
- Correction to the BMJ paper – 10 July 2020
- The (non) revision of the BMJ paper – 10 July 2020
- Time to correct the record
- Questions arising
- What has gone wrong here? How should we look at youth vaping?
- Advice to researchers using surveys to influence policy
1. Introduction
In June 2019, a controversial and high profile paper, Hammond et al. 2019 (BMJ), on teenage vaping and smoking in Canada, England and the United States from a highly regarded research team derailed progress on tobacco harm reduction in Canada. It contained the shock finding that not only had adolescent vaping risen sharply, but that adolescent smoking (past-30-day use) had shot up from 10.7% to 15.5% between 2017 and 2018. Given smoking is likely to be at least twenty times as harmful as vaping over the longer term, the reported rise in smoking was the real blow in this data. Unfortunately, the rise in smoking was an illusion – an error arising from inappropriate weighting. Eventually, though inadequately, it was corrected in the BMJ in July 2020.
The paper was trailed with officials and the media from late 2018 onwards, well before its publication. Its conclusion notes the significance of its timing in Canada’s policy development:
This discussion has come to the fore, as Health Canada and the US Food and Drug Administration contemplate measures to protect young people, including greater restrictions on advertising and promotion, flavors, and retail access to e-cigarettes.3031
31. Health Canada. Notice of Intent – Potential Measures to Reduce the Impact of Vaping Products Advertising on Youth and Non-users of Tobacco Products. Government of Canada. 2019 Feb 7. www.canada.ca/en/health-canada/programs/consultation-measures-reduce-impact-vaping-products-advertising-youth-non-users-tobacco-products/notice-document.html (accessed 15 Feb 2019).
I asked Canadian public health legend David Sweanor to summarise the deterioration in tobacco/nicotine policy starting with this paper. Here is his response:
Health Canada has lost the plot. Pushed by media hype about youth vaping that was hugely fed by the Hammond paper and a reality distortion field on EVALI, and urged on by abstinence-only advocates, any sort of strategy to achieve the announced goal of 5% smoking rates by 2035 has been lost. The compassion now shown on other drug issues is missing when dealing with people who smoke cigarettes, replaced by coercive measures and stigmatisation.
Lethal misinformation on relative risks has been left unchallenged and risk-proportionate regulatory measures evidently abandoned as any pretence of balancing the risks of continued smoking with concerns about limiting nicotine use by young people has been swept aside. Yet there is indisputable evidence of the horrendous health toll of perpetuating smoking and the way current policy directions will do that. There are also global success stories showing the substitution of non-combustible alternatives for cigarettes could achieve that 5% goal long before 2035. Health Canada leadership could still strive for that goal, but only if it is willing to hold its nerve and take a clear-sighted view of the evidence and a proportionate approach to risks.
In this post, we will follow the progress of this paper up to the point of its (wholly inadequate) correction.
2. The paper in the British Medical Journal – 20 June 2019
On 20 June 2019, the BMJ published data comparing vaping and smoking prevalence in Canada, England and the United States, comparing various measures of prevalence in 2018 to 2017.
Hammond D, Reid JL, Rynard VL, Fong GT, Cummings KM, McNeill A, et al. Prevalence of vaping and smoking among adolescents in Canada, England, and the United States: Repeat national cross-sectional surveys. BMJ 2019;365:l2219 [link]
The research is part of the $15 million ITC programme funded by the US National Cancer Institute that has been running since 2016. The overall programme is titled “Evaluating how tobacco control policies are shaping the nicotine market“. The subproject that produced this research is referred to in the BMJ paper as 1P01CA200512-01 and Sub-Project ID: 8133 and this sub-project is titled: “Vaporized nicotine product initiation among youth in the US, Canada, and England: methods to predict uptake and policy efficacy.”
Here is an extract from Table 2: Changes in prevalence of smoking and vaping between 2017 and 2018 among adolescents aged 16-19 years, by country, showing the shock data:
Yikes!! These are very substantial and alarming increases (from 10.7% to 15.5% is a 45% increase in one year). But are such steep rises realistic after years of decline? That should have rung an early alarm bell.
3. Publicity for the BMJ paper – 18 December 2018 (six months before publication)
The publicity for this paper started well before the paper was actually published. The BMJ paper was also heavily (and unethically in my view) trailed many months before its publication in the media and with key officials. For example, the national broadcaster CBC carried the following on 8 December 2018, six months before publication: Teen vaping in Canada has taken a ‘worrisome’ turn – New data suggests teen smoking rates in Canada are also rising:
As he tallied his latest data on vaping rates among Canadian teens, University of Waterloo Prof. David Hammond tried to find reasons not to believe his own research.
That’s because the results were troubling.
And it’s true. The results – to the extent that we were told about them – were troubling. The reason to be troubled was not especially the rise in teen vaping. Though that does carry political cut-through, it doesn’t have especially serious public health consequences. From a public health point of view, the alarming but still unpublished finding was that there had been a rise in teen smoking at the same time. Again, from the CBC report:
And even more disturbing — cigarette smoking in teenagers appeared to be rising for the first time in 30 years.
“There are also troubling findings on smoking rates and signs that progress in reducing youth smoking may have stalled,” he said, adding that there’s a need for more research to confirm his results.
“We all want these findings not to be true.”
As it turned out they weren’t true, so that’s good news I guess. But this false finding lingered from late-2018 to July 2020 – a critical period in the development of Canadian tobacco/nicotine policy.
4. Official Canadian data diverges sharply from BMJ paper – 25 June 2019
On 25 June 2019, less than a week after the publication of the BMJ paper, Statistics Canada published data on smoking prevalence by age:
- Statistics Canada. Table 13-10-0096-10 Smokers, by age group, 25 June 2019: [link]
- Statistics Canada. Table 13-10-0096-23 Current smoker, daily, by age group, 25 June 2019 [link]
This showed declines in adolescent smoking between 2017 and 2018 in Canada (see below). Alarm bells again.
5. BMJ rapid response and PubPeer comment on the official data – 9 July 2019
Just in case the official data had somehow escaped the attention of the authors of the BMJ paper, I set out the relevant data in two posts on 9/10 July 2019:
- a rapid response to the BMJ: Increases in smoking recorded in this study appear to conflict with official Canadian data, 9 July 2019 [link]
- a comment on the paper on PubPeer: Increases in smoking recorded in this study appear to conflict with official Statistics Canada data, 10 July 2019 [link]
Here is the data in a nutshell
Data published in this paper
In this paper, Table 2 shows marked increases in smoking in Canada between 2017 and 2018 among 16-19 year-olds:
- Smoked in past 30-days: increased 45% from 10.7% to 15.5%;
- Smoked in the past week: increased 57% from 7.6% to 11.9%;
- Smoked ≥15 days in past 30 days: increased 54% from 4.8% to 7.4%.
These (improbable) increases conflict with official data from Statistics Canada that were released shortly after the publication of the paper drawn from the Canadian Community Health Survey, with a representative sample of 65,000 [1][2]. The age groups reported by Statistics Canada do not exactly match that reported in Table 2.
Official data
However, the official data do suggest caution is essential in interpreting or acting on the findings in this survey:
Statistics Canada: Age group 12-17 years:
- Current smoking declined by 9% from 3.5% to 3.2%;
- Daily smoking declined by 18% from 1.1% to 0.9% (*estimated).
Statistics Canada: Age group 18-34 years:
- Current smoking – unchanged at 19.2%;
- Daily smoking – declined by 9% from 11.3 to 10.7%
In both cases, the corresponding author should be aware of the post. Alarm bells clanging!
In another BMJ rapid response, UK experts who run a substantial survey questioned the UK data: Jamie Brown, Smoking prevalence in young people in England, 2017-2018, 3 July 2019 [link]
In summary, whereas Hammond and colleagues reported modest increases in smoking prevalence among 16-19 year olds between 2017 and 2018, both the STS and APS reported numerical declines that did not support modest increases between 2017 and 2018 for 16-19 and 18-19 year olds, respectively.
Alarm bells!
6. Publication of 2019 data with a radical revision of 2018 data – 4 May 2020
On 4 May 2020, Dr Hammond and colleagues published 2019 data along with a revision of the 2018 data in JAMA Pediatrics. Note many of the co-authors were different to the BMJ paper.
Hammond D, Rynard VL, Reid JL. Changes in Prevalence of Vaping Among Youths in the United States, Canada, and England from 2017 to 2019. JAMA Pediatr. 2020 May 4 [link] ).
The paper includes the following chart – the dark blue and brown bars represent smoking and I have annotated the chart in red:
This new JAMA Pediatrics paper was in obvious conflict with the high-profile data published the year before in the BMJ. The graphic shows past-30-day smoking decline from 10.7% to 9.9%. It also shows a sharp decline in regular smoking in 2019 – consistent with the view that vaping can displace smoking. Given smoking is by far the greater problem behaviour, there is, in fact, a good news story in the data.
The JAMA Pediatrics paper was accepted for publication on 6 Feb 2020 and was presumably written, submitted and peer-reviewed somewhat before that date (the date of the original submission, revision etc are not given by JAMA Pediatrics). It likely that the authors were aware of this discrepancy in the 2018 data once they started to use the data from the official Canadian statistics that had been made available in July and it would be absolutely clear that the 2018 data presented in the 2019 BMJ paper was unreliable when the JAMA Pediatrics paper was submitted around the end of 2019 (I guess).
7. Technical paper – 5 May 2020
Hammond D. Reid JL, Rynard VL, Burkhalter R. ITC Youth Tobacco and Vaping Survey – Technical Report Survey: Technical Report Wave 3 (2019). University of Waterloo. Updated May 2020. [link]
This clearly lays out the change in Table 4 on page 12, which is good. Yellow highlighter in the original, my annotations in red. The large ellipse shows the dramatic revision in 2018 smoking prevalence for ever-smoking, past 30-day and past-week smoking frequencies. It also provides a more useful indicator for frequent smoking – smoking on ≥20 days in the past 30.
The other thing to note (see the small circles) is the rate of frequent smoking – in my opinion, the only real problem behaviour shown in this table. This fell sharply between 2017 and 2019 in the two countries, Canada and the US, that had larger increases in vaping, but increased in England. Given smoking is at least twenty times as harmful, this is an important change in the mix of what Professor Ian J. Irvine calls their “sin portfolio”. In this case to lower portfolio risk. It is far from clear in these data that England should be a role model for policies to implement in Canada and the United States, once youth smoking is considered.
The technical report contains an explanation for the change. The change was in the ‘weighting’ regime used:
In 2019, the US National Youth Tobacco Survey (NYTS) and the Canadian Student Tobacco, Alcohol and Drugs Survey (CSTADS) were released, which allowed us to enhance our weighting procedures. We used these surveys to calibrate to the trend over time for past 30-day smoking in the US and Canada. NYTS and CSTADS represent the national monitoring surveys in each country, providing nationally representative estimates for the US and Canada. Briefly, the percent change in the past 30-day smoking rate observed in the national monitoring surveys was applied to our Wave 1 estimates, providing Wave 2 and Wave 3 target rates for weighting; Wave 1 was not adjusted in any way.
Weighting explained: The specific survey sample responses may not be representative of the full population (i.e. age, sex, ethnic mix may differ). Weighting is used to convert the results found with the sample to reflect the results you would expect for the real population.
What is not well explained is how the original weighting and updated weighting created such a huge non-marginal change in the critical result. What could have changed so much in the structure of the population since the data they used for the original weighting used in the BMJ paper to cause the results to change so much when readjusted? Was it something else? I don’t have an answer to that question, but do the authors? [see informative comment from Thomas Lumley]
There are also questions about the response rates and retention rates for these surveys – to my untrained eye, they are very low – and they are not fully disclosed in either the BMJ or JAMA Pediatrics paper, other than via a reference to technical papers on Dr Hammond’s website. Feedback from data nerds would be welcome: the relevant technical papers are here.
8. Correction to the BMJ paper – 10 July 2020
On 10 July 2020, the 2019 BMJ paper was finally corrected. Sort of.
A correction notice was added to the paper visible on the BMJ website.
The correction itself is completely uninformative. It merely points to a data supplementary file with no indication of the nature and materiality of the error. The abstract for the correction is empty. Yet the correction changes one of the most important conclusions of the paper.
When the determined reader eventually reaches the supplementary file, it does describe the correction due to more appropriate weighting.
The effect of this reweighting was to reduce the 2018 estimates of smoking prevalence in Canada and the US to match the national trends observed in benchmark surveys. In the original paper in The BMJ, changes in past 30 day smoking prevalence between 2017 and 2018 in Canada were reported as 10.7% to 15.5% (a statistically significant increase), which was revised after reweighting to 10.7% to 10.0% (no significant change).
9. The (non) revision of the BMJ paper – 10 July 2020
So of course, the revised data published in the supplemental file was used to change the substance of the published paper. Only joking!
Despite the more accurate weighting generating a pretty fundamental change in the results and the complete reversal of a stand-out conclusion, the original 2019 BMJ paper appears entirely unchanged. Other than the correction pointing to the supplementary file, the text of the paper itself has not changed at all, as far as I can see.
In the abstract, the results still inaccurately say:
The prevalence of vaping in the past 30 days, in the past week, and on 15 days or more in the past month increased in Canada and the US between 2017 and 2018 (P<0.001 for all), including among non-smokers and experimental smokers, with no changes in England. Smoking prevalence increased in Canada (P<0.001 for all measures), with modest increases in England, and no changes in the US. (emphasis added)
The conclusion still inaccurately says:
Between 2017 and 2018, among 16 to 19 year olds the prevalence of vaping increased in Canada and the US, as did smoking in Canada, with little change in England. (emphasis added)
The results table 2 remains unchanged.
The discussion section discusses something that didn’t happen:
The increase in smoking among Canadian adolescents raises important questions about the association between vaping and smoking behavior. After several decades of steady decline, smoking among 15 to 19 year olds did not change significantly between 2015 and 2017, and no other national estimates are available for 2018.14 If the increase in smoking prevalence in Canada was directly related to an increase in vaping, similar increases in smoking would be expected among US adolescents, who reported a similar rise in vaping; however, no statistically significant changes in smoking were observed between 2017 and 2018 in the US. Estimates from the current study are closer to the 0.5% increase in past 30 day smoking observed in the National Youth Tobacco Survey study1617 than the 0.8% decrease recorded in the Monitoring the Future study in 2018.18 The greater increase in smoking in Canada might reflect emergent trends leading up to the legalization of non-medical cannabis in October 2018. (emphasis added)
I am mystified by this. We now have the main BMJ paper and abstract (June 2019) in conflict with the Canadian official data (July 2019), the JAMA Pediatrics paper (May 2020) and the supplementary data provided to the BMJ paper (July 2020) in the form of a correction to itself.
How is this going to stop the BMJ paper being cited in error – as it was, for example, recently cited by Australian hapless anti-vaping Emeritus Professors Chapman and Daube relying on the very point in the BMJ paper that is now understood to be wrong and reversed?
….and Canada, which, as shown in a paper co-authored by Borland, has seen not only a dramatic increase in youth vaping, but also the first increase in youth smoking in many years [14].
Chapman S, Daube M. Response to Mendelsohn, Borland and Hall’s ‘Could vaping help lower smoking rates in Australia?’ Vol. 39, Drug and Alcohol Review. Blackwell Publishing; 2020. p. 419–21. [link]
10. Time to correct the record
In my view, the record needs to be properly corrected, and that includes amending the abstract, methods, results, discussion and conclusions results of the BMJ paper so that they become consistent with revised and more realistic data, which they currently are not. To reiterate, I am not seeing a conspiracy here, just a reluctance to own the errors made and put them right properly leading to an absurd dissonance between the manuscript and its corrected data.
The BMJ paper (as published and accessed 23 July 2020) even has a transparency statement at the end of the manuscript:
Transparency: The lead author (DH) affirms that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained.
This is plainly no longer the case given the correction. The published manuscript is full of discrepancies with the revised data and what are now demonstrably false statements.
11. Questions arising
When it did it become clear that the weightings used in the 2019 BMJ paper were generating a spurious and highly controversial result suggesting adolescent smoking was rising when it wasn’t?
What was done and when to inform: (1) the co-authors of the BMJ paper and ITC project; (2) Health Canada; (3) the BMJ; of the changes generated by more realistic weighting?
What amendments are required to the main 2019 BMJ paper to bring it into line with the revised data?
When will the revisions be provided to the BMJ?
Would it be better to retract and resubmit the BMJ paper with the revised data?
12. What has gone wrong here?
I have concentrated on the error about smoking because that was the most egregious problem with this paper. I do not think the authors did anything deliberate or malicious and I will assume the weighting in the original paper was done in good faith – even if it produced what was discovered later to be a flawed estimate of smoking prevalence. The question is what happened and when once this became clear.
The paper also fits into a wider pattern of problematic “youth vaping epidemic” research – discussed in these two posts: The great America youth vaping epidemic. Really? and The US flavour ban: twenty things you should know. Changes in youth smoking and vaping in Canada (and elsewhere) need a nuanced interpretation that is missing from these reports. Through papers and media comments, Dr Hammond appears to have been building a case to support caps on nicotine strength in e-liquids and restrictions on marketing that would reflect the approach taken in the European Union. This is the case across several papers and media interviews. The BMJ 2019 paper summarises what is known and what the paper adds, clearly aiming to build this theory (annotated by me in red):
The problem is, the paper doesn’t actually support this conjecture in Canada. Table 5 of the BMJ paper shows that Juul is only the third most popular product in 2018, far behind the ‘Smok’ brand which is not constrained by EU regulation.
Obsession with Juul. The word ‘Juul’ is mentioned in the text of the BMJ paper a whopping 52 times, but ‘Smok’, the brand found to be most popular with youth in Canada and England, is mysteriously not mentioned at all (other than in the table above). This looks to me like the authors opportunistically joining the mob attack on Juul that was ongoing in the US at the time. There is nothing here that should single out Juul or its characteristics for special significance or a regulatory finding, but the paper uses it to advance the authors’ theory about high nicotine concentrations and nicotine salts.
The conclusion to the JAMA Pediatrics paper continues to advance this theory:
The increases in frequent vaping in the US and Canada are consistent with the increase since 2017 in the popularity of nicotine salt products, such as Juul, which have markedly higher nicotine concentrations compared with earlier generations of e-cigarettes.5 In England, e-cigarettes are subject to greater marketing restrictions and a maximum nicotine concentration of 20 mg/mL.
[…]
The extent to which the lower vaping prevalence in England is associated with the different market and regulatory environment warrants close consideration.
This is typical of tobacco control academic overreach. It starts with the assertion of an explanatory theory – that characteristics of the Juul product and its marketing caused the rise in youth use – though without demonstrating its validity from data presented in the paper. Because these things are regulated differently in Europe and England doesn’t have the youth vaping spike then these things could be the reason for the youth increase in North America. Then follows the implicit assumption that banning these characteristics will reduce youth use. The policy analysis needed to draw this conclusion is far beyond what has been done here – I discuss some concerns in the next section.
13. Advice to researchers using surveys to influence policy
Here is my advice to researchers (not just this group) doing this sort of work and hoping to inform policy with it:
- Never look at nicotine policy through the singular lens of youth use – adults matter (a lot). From a public health perspective, the real at-risk population is middle-aged and older smokers (40+). For them, e-cigarettes can provide a radical reduction in imminent and rising risk of serious disease. The Juul-type pod products have been highly successful at switching adult smokers to vaping. They use relatively high strength nicotine because they deliver low volumes of liquid (not necessarily more nicotine) and part of their appeal to adults is their small size, convenience and ease of use. If you restrict these, there may be a cost in disease risk to adults. In a political approach, it is sometimes possible to ignore adults and focus exclusively on youth. In an ethical public health approach, the full range of benefits and detriments to all populations should be considered.
- Never look at vaping without looking at smoking, including adolescent smoking. The publication of what turned out to be false data points showing increases in youth smoking in Canada allowed the authors to ignore the possible substitution of youth smoking by vaping and bring the political focus on to youth vaping. But a fall in teen smoking caused by vaping could be a real public health advantage coming from increased vaping – given that smoking is likely to be at least twenty times as dangerous. In fact, in 2019, Canada showed a sharp decline in adolescent frequent smoking not evident in England, leaving England with a higher proportion of adolescent frequent smokers. It is possible this was the health dividend from Canada’s rise in vaping. It’s not clear to me that the youth smoking/vaping situation in England is preferable to that in Canada, given the relative risks. The BMJ paper sets up an inadequate framing for policy – the point is that capping of nicotine strength is likely to do more to hurt adults and little to protect youth.
-
Rather than fuel the debate over trade-offs between less harmful nicotine alternatives for adult smokers versus increased initiation among adolescents, the findings should direct attention to regulatory measures that more selectively target these vaping products to adult smokers
-
- Always segment headline teen vaping figures by frequency of use and prior tobacco use. The ITC papers have provided some valuable insights in this respect. But we should be clear – what matters from a public health rather than Puritanical point of view is the frequent use of nicotine products – frequent use is a developing substance use pattern that may ultimately consolidate into frequent daily smoking (or vaping) and become hard to quit. If the frequent vapers are also more likely to have been or to become smokers, their vaping is likely to be a win for public health. Teens blowing vape rings at parties or messing around with vapes is not a cause for alarm or a reason for adults to lose their minds.
- Assess population harm by harm-weighting product use. Yes, it is easy to generate a moral panic around youth vaping. But what if youth vaping is just a reconfiguration of youth risk-behaviours (e.g from smoking to vaping). Vaping is among the least troubling of all youth risk behaviours? What if, overall, the ‘sin portfolio’ of youth is becoming less dangerous when it comes to nicotine, but much more dangerous when it comes to other risks – opioids, dangerous driving etc. What is the relative harm associated with one middle-aged adult denied smoking cessation via vaping versus one net additional youth who takes up vaping who would never have used nicotine? Unless researchers have a feel for that harm ‘exchange rate’, they have no basis for making a public health recommendation.
- Regulatory policies should never be pitched in narrowly-focused data papers. If you want to pitch policies, start with a designing the research programme that you would need to establish if the proposal would work and assess the full range of potential harmful unintended consequences. You cannot just tack a policy recommendation or giant hint, wink and nod in the direction you want to steer policymakers. Start with an economic model that at least allows the possibility that these products are substitutes – there is already good research showing this (e.g. see Mike Pesko’s e-cigarette research summary). Research would need to test the basis of preferences and behavioural motivation (i.e. is the preference a cause of the behaviour and they would not be doing it if their preference was not available? Then there is the question of intervention research. Would young people stop vaping and devote more time to homework and church attendance, or would they adjust their portfolio of risk behaviours – possibly to more risky behaviours? What about the commercial and black market response? Banning something does not make it disappear. After the frenzy of pearl-clutching about Juul in the United States and that company’s own self-flagellating withdrawal of flavoured products from many retail locations, we simply saw the rise of the Puff-Bar, a product that is illegal under the US system of regulation.
- Could adults please grow up about youth risk behaviour? In reality, it is impossible to hammer youth risk behaviours out of existence using regulation. We know from the war-on-drugs that even the harshest regulation – a prohibition – is of little use but comes with huge harms arising from black markets and use of the criminal justice system to address a public health problem. Yet, in the United States, over 20% of young people (at 12th grade) have been past-30-day cannabis users for more than two decades. Do researchers ever learn lessons about regulation from the actual experience of regulation and youth risk behaviours?
- Funding. This concern is not specific to this programme or project, but I believe the funding system creates a gigantic distortion of research priorities on youth risk behaviours. Large sums of are raised from user fees paid to the FDA and spent in federal research programmes. This project and many others are funded to look at nicotine products and, of course, that is what they do. But given a blank slate, a research approach to the risks facing young people would not take this single substance approach. It would differ in two distinct ways: (1) it would look at the evolving portfolio of risk behaviours in the round and try to detect positive and negative trends in movements within the risk portfolio and focus on mitigating the greatest harms for the greatest number; (2) it would take a person-centric, not substance-centric, view of youth risk behaviour, asking what is going on in their lives and working on the assumption that risk behaviours mainly arise from characteristics of the person and their circumstances.
Final note
I welcome discussion and feedback – positive or negative – and I promise to correct or amend anything in this blog that is unfair.
What is the best way to push BMJ for a real ‘correction’?
The best way is for the authors to correct the manuscript themselves. I’ll wait and see if that happens. If diplomacy fails, we can try the alternatives.
The reason for a big change with the revised weighting wasn’t a change in the population structure, precisely. It was a change in the variables used to define the weighting target. The new weighting used ‘change in national estimated smoking rate’ as one of the target weighting variables; the old one didn’t. That means the new figures were forced to match national data on smoking trends (which is why they now do match national data on smoking trends).
(p11 of the report: Briefly, the
percent change in the past 30-day smoking rate observed in the national monitoring surveys was applied to our Wave 1
estimates, providing Wave 2 and Wave 3 target rates for weighting; Wave 1 was not adjusted in any way.)
It’s not clear *why* the smoking rate estimates were so badly off before the change, but since it’s clear that they *were*, it’s not surprising the new weighting fixes it.
Hi Thomas, thanks for this feedback. I have put a link to your comment in the relevant part of the blog. I guess I’m mystified by how the original weighting was done and no wiser about how this apparently large discrepancy was generated.
I’m sure the original weighting was done in good faith and with whatever was available to the authors at the time. The issue for me is how the difference was recognised and addressed, and how quickly.
Clive
Give em hell Clive.
Wonderful analysis! It reminds me of the runaround Dave Kuneman and I got when we submitted a well done paper flatly contradicting the conclusions of Glantz’s Helena smoking ban BMJ research from 2003.
Quick question:You mention substantial “user fees paid to the FDA”. Is that simply a euphemism for taxes that they couldn’t legally get passed by the legislature? And do you have any estimate on what they amount to per item or(preferably) pack-equivalent?
-MJM
The user fees are part of the Tobacco Control Act. Section 919 sets out a schedule of fixed payments from the industry to FDA and how share for each company is calculated.
After tapering in for 10 years, this is now $712 million per year in perpetuity. This funds the FDA’s Center for Tobacco Products, including the funding it provides via NIH for “Regulatory Science”. Strictly it is not a tax, but a payment for the ‘service’ of being regulated. Economically and from consumer perspective it functions like an ad valorem tax.
“Strictly it is not a tax, but a payment for the ‘service’ of being regulated. Economically and from consumer perspective it functions like an ad valorem tax.”
Yep. Same deal as our Master Settlement Agreement. Back in 1998 cigs were priced at perhaps $1.50 to $2.00 per pack in most of the country and there was STRONG resistance among legislators to adding even a ten cent tax on that. The MSA added 50 cents a pack, SUPPOSEDLY as a “punishment” to Big T and a way for them to pay for smokers’ health care…
… EXCEPT that I believe it is explicitly written into the MSA that Big T was *NOT* even ALLOWED to pay it: the cost had to be passed directly to the smokers as a pack surcharge! The whole deal was arranged by a compact of 40ish individual State Attorney Generals banding together and threatening to sue Big T out of existence unless it agreed to act as a tax collector for a not-tax! (btw… interstate “compacts” are explicitly unConstitutional but they just did it anyway, and Big T stupidly went along with it because it had that guarantee that smokers, and not execs and shareholders, would be forced to pay the money.
::sigh::
Re the flawed paper: if anyone wants to see the BMJ’s *past* reluctance to behave as a responsible medical journal in this area, just visit https://web.archive.org/web/20140326031547/http://acsh.org/2007/07/a-study-delayed-helena-mts-smoking-ban-and-the-heart-attack-study/ to see the game they played. They basically admitted that the smoking ban study was total b.s. but said there was no “need” to publish our contrary research since our conclusion, i.e. that the ban study was faked nonsense, was already “generally known”!
Seriously. It’s an incredible tale overall. And in the five or so following years there were at least three studies from sources like Stanford Univerisity and the RAND Corporation that were very similar to ours and which were published in various (non-medical) journals.
Clive, that truly IS a beautiful analysis reinforcing the BMJ’s general longstanding irresponsibility in this area.
– MJM
Publishing papers that contain honest (not intentional) errors is common in all disciplines. All researchers have had their share of papers with errors. It is the responsibility of authors to write an erratum (a second very brief paper with the correction).
However, the authors’ responsibility does not end there, they have to cite the erratum wherever and whenever they cite the first paper and when they interact with peers and non-scientific outlets (media, etc). It is crucial that the authors take upon themselves the task of informing the community (and society) about the error and its correction.
There should be also an institutional response to this. The search engines must also place the erratum together with the original paper. This is done for example in the INSPIRE High Energy Physics search engine, perhaps the most popular in physics (but Google Scholar doesn’t do it).
I have had my share of papers with honest mistakes. This is the latest example:
https://iopscience.iop.org/article/10.1088/1475-7516/2016/03/012
This is the erratum we published in the same journal
https://iopscience.iop.org/article/10.1088/1475-7516/2016/06/E03
If you go to the INSPIRE engine and look for me you will find the paper cited (the # 15 top-down) and see that the paper is cited with its erratum. I always cite the erratum whenever I have presented this paper’s results in congresses and lectures.
I don’t know how this works in medical journals. I don’t know if the authors of the Hammond et al paper have written an erratum. I don’t know if PubMed or other search engines cite the correction as erratum (it seems they don’t, they seem to be citing it as “supplement”).
Hopefully whoever searches in the future for the Hammond et al paper will be IMMEDIATELY informed about the error. Unfortunately, it seems to me (I might be mistaken) that the authors of the Hammond et al paper unintentionally made an honest mistake that that suits an institutional political agenda dominated (at least in the USA) by anti-vape influential tobacco controllers. Thus, informing the community about this error might not be as straightforward as in physics.
“I do not think the authors did anything deliberate or malicious and I will assume the weighting in the original paper was done in good faith”
Charitable.