This paper has been doing the rounds, claiming that lockdown was useless (the source of the 0.2% effect of lockdown claim). Dozens of people have asked my opinion of it, so here we go:
In my opinion, it is a very weak review that doesn't really show much, if anything 1/n
Conversation
Replying to
2/n The paper is a systematic review performed by three very highly-regarded economists who have also been extremely anti-lockdown since March 2020. You can find it here: sites.krieger.jhu.edu/iae/files/2022
15
69
717
3/n As others have noted, this is a "working paper", which essentially means it's not peer-reviewed and reflects only the opinions of the three authors named
7
82
765
4/n The first issues with the review itself are very basic - the inclusions/exclusions
None of the criteria make much sense to me
3
37
550
5/n For example, all papers with modelled counterfactuals are excluded. Because this is the most common method used in infectious disease assessments, this has the practical impact of excluding most epidemiological research from the review
15
85
765
6/n The authors claim that they only include studies using a "difference in difference approach", but according to other economists, well...
Quote Tweet
Meta-shmeta analysis. They claim they find that lockdowns reduced mortality in Europe and U.S. only by 0.2%. After browsing through their methodology and results though, it's obvious they aren't doing what they claim they're doing and their analyis is deceptive. /1
Show this thread
3
47
555
7/n So, the included studies certainly aren't representative of research as a whole on lockdowns - not even close. Many of the most robust papers on the impact of lockdowns are, by definition, excluded
3
43
564
8/n We can see the impact of this in the main table of results. This is the source of that 0.2% figure you might've seen thrown around
This is just a useless meta-analysis
1
39
454
9/n Why is it useless?
Well, let's look at the studies here
Bjørnskov = Oxford stringency index correlated to crude death reports
Shiva/Molana = Oxford stringency index correlated to crude death reports
Stockenhuber = well, you get the gist
2
33
387
10/n The only differences in these studies are the countries included, timing, corrections for confounding, and lags. Also some modelling differences
1
22
346
11/n One study that's noticeably absent is the Hale et al paper which PRODUCED THE OXFORD STRINGENCY INDEX ON WHICH ALL OF THIS IS BASED
They estimated a massive reduction in death due to lockdown
4
82
619
12/n If you've got 7 papers that take the same databases and manipulate them in different ways, it doesn't really make sense to calculate a mean weighted by standard error and call that the result. It's just bizarre
3
51
631
13/n But it gets even weirder. If you look at the model, almost the entire weighting is based on this paper, Chisadza et al
But Chisadza et al found a BENEFIT for lockdowns
5
41
501
14/n Indeed, the authors of this paper have publicly disagreed with the review, and accused the review authors of having a predetermined conclusion when writing the paper
Quote Tweet
I spoke to the author of the paper on whose research this entire meta-analysis was based, but who reached a diff conclusion.
She said: "They already had their hypothesis. They think that lockdown had no effect on mortality, and that’s what they set out to show in their paper." twitter.com/whippletom/sta…
3
58
530
15/n If you look closely, it seems that there are similar issues with quite a lot of the included research. For example, in table 5 much of the aggregate model is based on this paper...
1
24
354
16/n ...the senior author of which has agreed that you can't really interpret their regression estimates in this way
Quote Tweet
Replying to @econoflove @AndreasShrugged and 3 others
That also makes sense to me, which begs the question how the review came to an interpretation that they consider meaningful. Given that your paper makes up the majority of their secondary estimate of effect, it's a pretty important question!
1
23
353
17/n Another included paper found that significant restrictions were effective, but is included in this review as estimating a 13.1% INCREASE in fatalities. The maths used to derive this is pretty opaque
2
25
359
18/n All of this adds up to a very weird review paper. The authors exclude many of the most rigorous studies, including those that are the entire basis for their meta-analysis in the first place
1
39
422
19/n They then take a number of papers, most of which found that restrictive NPIs had a benefit on mortality, and derive some mathematical estimate from the regression coefficients indicating less benefit than the papers suggest
1
27
342
20/n There's no real rating of the potential for bias in the included research (the review only uses these 4 really bizarre 'quality dimensions' which...well)
5
22
326
21/n All of this together means that the actual numbers produced in the review are largely uninterpretable. The "quality" assessment is meaningless, and the numbers themselves not really correct
1
24
353
22/n In reality, the impact of "lockdowns" is very hard to assess, if for no other reason than we have no good definition of "lockdown" in the first place. The authors of this review define a "lockdown" as LITERALLY ANY INTERVENTION
2
52
461
23/n However, this is certainly not how the analysis is actually done. In most cases, it seems the authors have taken estimates for stay-at-home orders as their practical definition of "lockdown" (this is pretty common)
1
20
313
24/n And honestly, I'd agree that the evidence for marginal benefit from stay-at-home orders once you've already implemented dozens of restrictions is probably quite weak. One of the included studies, Spiegel et al, makes this point rather well
1
26
313
25/n That being said, if we consider "lockdown" to be any compulsory restriction at all, the reality is that virtually all research shows a (short-term) mortality benefit from at least some restrictions
5
30
337
26/n You can even see this IN THE REVIEW. The authors found benefits for compulsory facemasks, business closures, border closures/quarantine, and school closures. Some of the benefits were very large!
2
53
387
27/n Imo if we define "lockdown" as "the marginal benefit of stay-at-home orders on top of many other restrictions", it's probably fair to argue from this paper that the benefit might be quite small
1
20
303
28/n Indeed, that's been shown before, and is quite a reasonable position based on the evidence! The -0.2% figure is pretty meaningless, but the general idea is not totally wild
1
14
248
29/n If we define "lockdown" as "any compulsory restriction against COVID-19", however, this paper actually shows quite the opposite, that many compulsory restrictions are very effective at controlling the disease
3
27
370
30/n All of this comes with the huge proviso that the estimates in the paper probably aren't very useful, and the specific numbers are likely to be quite a bit off
2
15
269
31/n As a minor addendum, I would also note that I personally agree that a lot of people originally underestimated the impact of voluntary behaviour change on COVID-19 death rates - it's probably not wrong to argue that lockdowns weren't as effective as we initially thought
9
20
324
Show replies