Using models disconnected from reality to justify huge ‘endgame’ interventions

In this blog, we take a look at modelling used to justify ‘endgame’ legislation under discussion in New Zealand. Among other things, this would reduce nicotine in cigarettes and other tobacco available through legal channels to minimal levels. We find the modelling and data assumptions bear no relation whatsoever to the underlying processes and the effects that such legislation would trigger. Deep cuts in smoking are assumed as inputs to the model. Unsurprisingly, therefore, the model results show deep cuts in smoking, and this determines the resulting health benefits. But the assumptions have no grounding in reality and misinterpret and misuse trial findings.

• Background
• Regulating to reduce nicotine in cigarettes
• Modelling the impacts of a law to reduce nicotine in cigarettes
• A critique of the modelling
• An overview of our critique
• Summary
• Recommendations
• Analysis (abridged)
• Conclusion
• The fatal flaws
• The wrong way to do public health

Background

The New Zealand government and parliament are debating so-called “endgame” legislation that would bring in three quasi-prohibitive measures. The bill:

1. significantly limits the number of retailers able to sell smoked tobacco products;
2. aims to prevent young people from taking up smoking by prohibiting the sale of smoked tobacco products to anyone born on or after 1 January 2009;
3. aims to make smoked tobacco products less appealing and addictive by reducing their nicotine content and controlling other ingredients.

I made a brief response to a New Zealand Parliament Health Committee consultation on these, explaining my unease – here. In this blog, I’m focussing on the plan to denicotinise tobacco as this seems to dominate the effects shown in the modelling.

Regulating to reduce nicotine in cigarettes

Regular readers will know I am sceptical about the value of regulation to make deep reductions in nicotine in cigarettes – not because I like cigarettes, but because I think such policies are flawed:

• the approach is excessively coercive and not the right way to intervene in personal behaviours. Public health should be about helping people do what works for them to make their lives better, not imposing it on them. More on this is at The wrong way to do public health below.
• a grandiose gesture that would suck up resources and political capital but form a barrier to more patient and respectful progress, especially with disadvantaged groups.
• likely to fail as a result of numerous unintended consequences, including illicit markets and a poor understanding of how it will work in practice at an individual and community level.

You can read my views here: Twenty reasons to be sceptical about rules lowering nicotine levels in cigarettes – and what to do instead.

Modelling the impacts of a law to reduce nicotine in cigarettes

But what about some of the science used to justify these policies? New Zealand’s Ministry of Health commissioned modelling of the proposed measures, and this modelling has been considered by the Cabinet and incorporated into the regulatory impact assessment for the proposed law.

The modelling was led by a team from the University of Melbourne with a mixed team of New Zealand and Australian academics. It was published as a pre-print, presumably in part to invite comment.

Ouakrim DA et al. 2022 published as a MedRxiv pre-print.

This, in turn, draws on the modelling of denicotinisation done earlier in the year led by the University of Otago tobacco control group in New Zealand:

Wilson N et al. 2022 published in the New Zealand Medical Journal.

The Wilson et al. modelling draws heavily on a randomised controlled trial undertaken in New Zealand in 2009-10 in which Very Low Nicotine Content (VLNC) cigarettes were added to a standard smoking cessation intervention used by the NX Quitline:

Walker et al. 2012. published in Addiction.

To understand what the Ouakrim et al. modelling has done, in particular, the spectacular assumptions and their basis, the reader needs to look at all three of these papers in detail. The last two are paywalled and inaccessible to most non-academic readers, so policymakers and legislators are likely to find it hard to verify the assumptions. We hope our review will help with that, and we cite relevant sections of the papers.

A critique of the modelling

We have just published a critique of this modelling used as part of the justification for the proposal to introduce regulation in New Zealand to denicotinise all tobacco that is legally available.

Clive Bates, Ben Youdan, Ruth Bonita, George Laking, David Sweanor, Robert Beaglehole, Review of: Tobacco endgame intervention impacts on health gains and Māori:non-Māori health inequity: a simulation study of the Aotearoa-New Zealand Tobacco Action Plan, Qeios, 8WXH0J, November 2022. https://doi.org/10.32388/8WXH0J

This is a post-publication peer review of modelling undertaken by academics from Australia and New Zealand to estimate the effects of removing nearly all the nicotine from cigarettes and other tobacco products that would still be available in New Zealand.

Also, see the press release from ASH New Zealand, which initiated our review: Assumptions behind plans to remove nicotine from cigarettes “significantly flawed” 7 November 2022.

An overview of our critique

The full review and citations are here. This contains the summary, recommendations, highly abridged analysis, and conclusions.

Summary

Building on recent progress towards the New Zealand Smokefree 2025 goal, the Government plans to introduce tobacco control legislation giving ministers powers to implement three significant new policies:

• a steep reduction in the number of retail outlets that can sell tobacco;
• a ‘smokefree generation’ proposal that would make it illegal to sell tobacco to anyone born after a certain date, and;
• regulations to remove most of the nicotine from tobacco to reduce its appeal and addictive effects. 

In preparation for this legislation, the Ministry of Health funded academics from Australia and New Zealand to model estimates of the likely impact of these measures, especially their contribution to achieving the Smokefree 2025 goal. 

The modelling, published as a preprint, Ouakrim et al. (2022) ,is the subject of this review.  It focuses on the modelling of the denicotinisation of tobacco because, according to the authors, it has the greatest impact.

A number of significant flaws have been identified. The modelling is based on a fundamental and incorrect assumption that denicotinisation would reduce smoking by 85% over five years compared to business-as-usual. This draws on an earlier modelling paper, Wilson et al. (2022), supplemented by other literature and expert opinion.

The assumption, used as a key input to the model, is derived from a misinterpretation of a well-conducted randomised controlled trial of smoking cessation interventions that included very low nicotine content (VLNC) cigarettes in New Zealand in 2009-10, Walker et al. (2012).^[3]

The problem is that the trial design bears little relation to a population-wide denicotinisation regulatory intervention and its findings are not at all transferable to a model of the legislation.

• Volunteers who had already called the Quitline were given pharmacological and behavioural support;
• The intervention group were also given free VLNC cigarettes and instructed to smoke them if they wanted to;
• The trial intervention lasted only eight weeks, and its impact was assessed at six months.
• The trial does not include the most likely responses to the denicotinisation measure: switching to vaping, accessing an expanded illicit market, or workarounds by consumers or producers.

The results of the trial indicate that the 7-day abstinence quit rate at six months increased to 33% in the group with access to VLNC cigarettes, compared to 28% in the “usual care” control group, a 5% increment.  The modelling, however, implicitly assumes that:  

• All people who smoke – including people with no interest in quitting or who are unable or unwilling to access pharmacological and behavioural support – would achieve this rate of quitting;
• The same quit rate would apply if people had to purchase denicotinised cigarettes, rather than receive them free of charge.
• All of the quit rate, 33%, could be attributed to the introduction of a population-wide denicotinisation policy and this quit rate would compound over five years, deriving, erroneously, an 85% reduction in smoking prevalence; the trial provides no basis for assuming a regulatory intervention will have this smoking cessation effect after one year or that it will repeat year after year.

Finally, the modelling makes unrealistic estimates of the implementation timetable, transitional arrangements, and the effects of stocks and hoarding. In doing so, it greatly exaggerates any likely impact on the 2025 targets.

In conclusion, the modelling on which the proposed legislation is based is seriously flawed:  

• It makes ill-founded assumptions based on a misinterpretation of a smoking cessation trial in which denicotinised cigarettes were provided as an enhancement to standard smoking cessation interventions to people who were already making a quit attempt. 
• It does not reflect the real-world dynamics of the population-wide regulatory intervention it is supposed to represent.
• It fails to take into account illicit trade in regular tobacco and other “workarounds”. This could be substantial and must be incorporated into any modelling of the denicotinisation measure.
• Modelling the legislation for policymaking purposes should more accurately reflect the real-world processes involved (e.g. illicit trade, workarounds, switching to vapes) and place greater emphasis on transparency of the assumptions used, sensitivity testing and scenario analyses.  

Recommendations

We recommend that the government reconsiders its confidence in the policy assessment and impact analysis that underpins Cabinet support for denicotinisation. We note that further forecasts of impact will not be reliable or informative at this stage.

We recommend that the Government should require that:

• Future modelling should focus on testing a range of scenarios in relation to quitting behaviours, switching to smokefree products, and access to illicit tobacco products;
• A more focussed examination be undertaken of plausible unintended consequences of the legislation and the impact of trying to impose abrupt non-voluntary smoking cessation on the whole population;
• Further analyses examine the optimum timing for introducing these measures, for example, once smoke-free alternatives are more widely accepted and used voluntarily as alternatives to smoking.
• Such analyses build on recent progress towards the Smokefree 2025 goal by strengthening existing measures which seem to be working, including a broader and more concerted push to encourage people who smoke to change to smoke-free alternatives.

Analysis (abridged)

[the full analysis is available in the Qeios Review, but it is based on identifying and demonstrating ten problems with the Ouakrim et al. modelling.]

1. The authors have built a dramatic expected decline in smoking into their model as an input
2. Critical assumptions and the literature used are not transparent, and authors rely on their own expert judgement, introducing risks of bias 
3. The Walker et al. smoking cessation trial bears no relation at all to the effect of imposing a population-wide nicotine regulation covering all smokers.
4. The results presented by Walker et al. show a small incremental effect from the denicotinised cigarettes, not the 33% used by Wilson et al. 
5. Denicotinised cigarettes were given free to smokers as part of a trial, but in real life, smokers would have to pay and are unlikely to use these products
6. The authors assume that the smoking cessation effects estimated in year 1 will continue and accumulate in subsequent years  
7. The authors do not have a transparent approach to illicit trade and appear to have ignored it
8. The modellers should treat the denicotinisation measure like a de facto ban of regular tobacco rather than a smoking cessation measure
9. The modellers do not use a realistic approach to timing and transition
10. The modelling does not even try to reflect the real-world dynamics of a market intervention to denicotinise tobacco

Conclusion

The statistician George Box coined the phrase “All models are wrong, but some are useful” and followed this with the obvious corollary “Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful.”. The Ouakrim model is not useful, not because it is merely wrong, but because it does not even attempt to represent the underlying dynamics that would emerge following the implementation of the proposed legislation. 

The input assumptions used in the model determine the positive outcomes presented by the authors. In particular, the modelling inappropriately uses an input assumption that smoking will decline by approximately 85% in five years following a denicotinisation rule. The basis for this assumption is opaque and cannot be justified by the findings of the trial on which it is largely based. 

Further, no information is provided for the evaluation of the expert judgements used in the modelling; how did they come to their conclusion that denicotinisation would lead to an approximate one-third quit rate in the first year after mandatory denicotinisation, which is then sustained for a total of five years?

The modelling is arbitrary and wholly unreliable as a guide to the policy impact. The measures will have consequences, both intended and unintended, but this modelling provides no useful information about what these could be. 

We recommend that any modelling of this legislation should more faithfully reflect the real-world dynamics of the proposed population-wide measure and likely market response.  We suggest that it is compared to a stronger intervention to encourage switching to smokefree products, rather than business-as-usual.  

End of overview

The fatal flaws

The wrong conceptual model. Even though modelling is usually wrong, it is sometimes useful. Normally, modellers do at least try to reflect the likely underlying processes they are trying to simulate. Not so, in this case. In this case, the likely dynamic will not be the mass uptake of VLNC cigarettes (who wants to buy nicotine-free cigarettes?) or even mass smoking cessation interventions. It will be a mixture of illicit trade in regular cigarettes and other nicotine tobacco, workarounds like renicotinisation or home growing, and switching to vaping or other smoke-free products (legal and illegal). The reality will be messy, and it will evolve over time as the tobacco market reconfigures to adjust to the regulation. The three papers involved do not address the most plausible reality. The modelling sees the world through the eyes of controlling technocrats, not the chaotic flexing of a disrupted market. It reinforces an illusion of control that will likely be absent in practice.

Better estimates are not possible. We, the critics, do not know if the estimates made by Ouakrim et al. are likely to turn out right, or to be a huge overestimate or an equally huge underestimate – and neither do they. This is because the modelling is wholly uninformative and uses (and, in our view, misuses) data drawn from circumstances that bear no relation to the conditions that the modelling is supposed to simulate. That cannot be fixed with a new set of arbitrary but baseless assumptions. The numbers might turn out to be right, but that would be right in the same way that a broken clock is right twice a day.

Expert opinion without expertise. Trial data has been supplemented with ‘expert judgement’. However, there is no expertise in this area, as nothing like this has been attempted in a whole-population marketplace. There is no relevant literature to draw upon other than other guesswork by other experts, also operating in the dark.

The wrong way to do public health

Finally, I should say that I do not think a measure like this should stand or fall on modelling, and certainly not this modelling. The deeper question to me is whether governments should approach public health in this way: by passing laws that force a quite intrusive change on people who may not want it – even if lawmakers think it is for their own good. This is outside the scope of our review of modelling, of course, but it is a significant political and policy question.

There are cases where compulsion in public health is justifiable – for example, to control infectious diseases (COVID-19), where one person’s actions can harm another (gun control), or where large benefits can be attained for minimal intrusion and the law functions as a “nudge” or normalisation (seatbelts).

But should a government really try to force people to stop smoking by, in practice, taking away their cigarettes? Isn’t that excessively paternalistic and infantilising? Wouldn’t it be more dignified and respectful to help people to quit or to encourage a switch to smoke-free alternatives? I’m not taking a crude anti-government or extreme libertarian perspective. But the degree of coercion in this measure seems to me to cut straight across accepted principles of adult autonomy and informed choice.

Some will raise “addiction” as a kind of trump card that justifies denying any sort of agency or volition on the part of people who smoke. I’m just not buying that. Nicotine dependence is not some sort of zombified state of helplessness. Many people do overcome nicotine dependence, and plenty of advice and help is available. Smoke, not nicotine, is the main health problem. For those who want to continue to use nicotine or would find that easier than quitting, switching to vaping products, by choice and through encouragement, might be a better way. But what about the people who don’t want to quit or switch or feel they cannot? How will they be made to feel?

My view, and it is a subjective position, is that the “enabling state” is a better model for the government’s role in tobacco policy than the “coercive state” as far as it applies to consumers as citizens.