India’s Objections to WHO Study on COVID-19 Mortality Assessment Appear Untenable

The New York Times, dated April 16, reported that the release of a report by the World Health Organisation (WHO) estimating the global and country-wise excess mortality from COVID-19 had been delayed for months because of objections from India. 

Excess mortality refers to the number of deaths from all causes during a crisis above and beyond what is expected to be seen under 'normal' conditions.

Reportedly, the COVID-19 excess death toll in India is estimated to be around 4 million or about eight times the current official figures of 0.52 million. The estimate of 4 million deaths is in line with many studies that have estimated excess deaths due to COVID-19 (such as Anand, Sandefur and Subramanian, Christophe Z. Guilmoto, Murad Banaji and Aashish Gupta, Prabhat Jha et al., Lancet study, Tushar Gore and Viral V Acharya, among others) 

In February 2021, the WHO, in collaboration with the United Nations Department of Economic and Social Affairs (UN DESA), set up a Technical Advisory Group (TAG) on COVID-19 Mortality Assessment to know the scope and scale of the epidemic to shape countries' response and recovery efforts. The TAG comprises about 33 members – scientists and experts of international repute. There are about 40 observers and a nominated point person from each Member state to engage with the organisation. 

A report by Devex alleges that a TAG member, on the condition of anonymity, mentioned that the Indian government had asked for the estimates to be published "10 years later". 

India's Objections With the Methodology 

In response to The New York Times report, the Indian Health Ministry issued a statement on the same day. It stated, "India's basic objection has not been with the result (whatever they might have been) but rather the methodology adopted for the same."

India alleges that the model treats all countries with a 'one-size-fits-all' model. It argues that India is a big country, and owing to its large population and geographical area, it should be studied differently from the smaller countries. India also has objections related to certain variables that the WHO study includes, such as the inverse relationship between temperature and average deaths. It says that high variability in India's test positivity ratio was not considered in the model and that age-sex distribution for India is an extrapolation based on other countries' distribution.     

The specific objections outlined in the statement include - 

• The statistical model projects estimates for a country of geographical size & population of India and also fits in with other countries with a smaller population. Such a 'one-size-fits-all' approach and models, which are true for smaller countries like Tunisia, may not be applicable to India. 
• India's age-sex distribution of predicted deaths was extrapolated based on the age-sex distribution of deaths reported by four countries (Costa Rica, Israel, Paraguay and Tunisia).
• The model assumes an inverse relationship between monthly temperature and monthly average deaths, which does not have any scientific backing to establish such a peculiar empirical relationship. 
• The test positivity rate for COVID-19 in India was never uniform throughout the country at any point in time. But, this variation in COVID-19 positivity rate within India was not considered for modelling purposes. 
• The present modelling exercise seems to be providing its own set of estimates based on another set of historic estimates while disregarding the data available with the country. It is not clear why GHE 2019 has been used for estimating expected deaths figures for India, whereas, for the Tier 1 countries, their own historical datasets were used.
• Of the covariates used for analysis, a binary measure for income has been used instead of a more realistic graded variable. Using a binary variable for such an important measure may lend itself to amplifying the magnitude of the variable.

WHO's Position 

Although the WHO or other experts have not given a point-by-point rebuttal to India's objections, pending the release of the final report, in a general explainer, the WHO specifies that the model adopted for estimating excess deaths is not a 'one-size-fits-all' approach. It mentions that the process considers the specificity of countries (e.g., income level, reported COVID-19 deaths rate, test positivity rate, containment index) while ensuring global comparability.

Responding to the Indian Health Ministry's statement, Prof. Jon Wakefield, Faculty, School of Public Health, University of Washington, USA, who is also a member of WHO's TAG, posted a tweet on April 18, saying, "Released in response to the NYT article, the Indian MoH statement on the excess mortality method we used for India is inaccurate."

Wakefield shares an abstract from a paper describing the model developed to estimate excess mortality for WHO that responds to many objections raised by India. 

It explains that for a small number of countries for which national all-cause mortality (ACM) data are not available, which includes India, ACM data from sub-regions is taken. For India, data from up to 17 states and union territories is taken over the pandemic period, but this number varies by month. 

Further, for India, the sources used for the registered number of deaths at the state and union-territory level include either information reported directly by the states through official reports and automatic vital registration or by journalists who obtained death registration information through Right to Information requests. They stressed that "for India, the global predictive covariate model is not used, and so the estimates of excess mortality are based on data from India only."

Government's Massive Under-reporting of COVID-19 Deaths 

Over the last two years, since 2020, while the pandemic has devastated the globe, there has been a controversy over the officially reported number of deaths. In India, during the second wave of the COVID-19 pandemic in 2021, people witnessed distressing scenes of floating corpses in the Ganges, long queues outside crematoriums, over-stretched crematoria with several pyres burning simultaneously and riverbeds strewn with numerous unmarked graves.

But the officially reported figures of deaths did not adequately cover the tragedy.  

To understand the magnitude of this discrepancy, many experts, epidemiologists, modellers, and journalists attempted to measure and arrive at the closest possible figures for COVID-19 deaths. Different methodologies were used, but the results were similar. Almost all of these studies, some of which were covered by NewsClick last year, showed that there was massive under-reporting of deaths by the government – 

1. A study by Anand, Sandefur and Subramanian (Centre for Global Development) arrived at three different estimates of the excess number of deaths using three different methods, which put the death toll at 3.4 million, 4 million and 4.9 million, respectively, by around June 2021. This was nearly eight to 12 times higher than the official figures.
2. A significant study by Christophe Z. Guilmoto, which did not use the excess mortality method, estimated India's mortality at about 2.2 million by late May 2021 – nearly five times the official count. 
3. Murad Banaji and Aashish Gupta estimate excess deaths between April 2020 and June 2021 to be about 3.8 million deaths, with more optimistic and pessimistic bounds ranging from 2.8 million to 5.2 million - Excess deaths are between seven- and 13-times higher.
4. Prabhat Jha et al. estimated that India's cumulative COVID deaths by September 2021 were six to seven times higher than reported officially. 
5. A more recent Lancet study, COVID-19 deaths between January 1, 2020, and December 31, 2021, states that at the country level, the highest numbers of cumulative excess deaths due to COVID-19 were estimated in India at about 4.07 million – nearly eight times more than the official figures.
6. A study by Tushar Gore and Viral V Acharya, using the patient flow model in eight districts, showed an underreporting bias for measuring COVID-19 deaths between three to nine times.
7. The Hindu newspaper published a series of reports capturing excess deaths across select states using the civil registration system (CRS) data. The excess deaths overshot the officially reported deaths by a factor ranging from 1.5 times to nearly seven times. 
8. Rukmini S, reporting in IndiaSpend, analysed CRS data for five states, Andhra Pradesh, Bihar, Kerala, Madhya Pradesh and Tamil Nadu March 2020 to May 2021 and found that most states, except Kerala, reported excess deaths in the range of 0.1-0.2 million during the pandemic. These state-level excess deaths in 2021 range from about 40 times in Madhya Pradesh, 33 times in Andhra Pradesh, to only 1.7 times in Kerala.  

WHO Study Appears Sound: Experts

To understand whether India's objections hold any water, NewsClick reached out to a few experts, some of whom have rigorously studied and estimated the COVID-19 deaths. 

According to Prof. Christophe Z. Guilmoto, Senior Fellow, The Centre de Sciences Humaines (CSH), Delhi, "The WHO methodology seems rather reasonable to me, as it is more or less based on a rule of thumb. Its Bayesian framework gives a robust way to compute the confidence intervals for the estimates. It is based on the excess mortality estimation, a standard procedure that has long been used in India, for instance, for epidemics during the colonial period."

Pointing to the various studies undertaken, Guilmoto argues, "We already have quite a few different estimates that all point to about 30-40 lakhs COVID deaths in India. My estimates point to about 2.2 million deaths. These make no use of the excess mortality method and therefore offer an independent confirmation of the range of the total number of deaths. This reinforces the sense that only one out of 6-8 deaths were correctly registered in India, though this ratio varies across states."

"The WHO's estimates of pandemic mortality in India - as reported in the New York Times - are very consistent with previous studies from India," says Dr Murad Banaji, Mathematician and Disease Modeller, Middlesex University, London. 

"Government of India's objections often begin with a grain of truth - that there are uncertainties and methodological difficulties - and use this to imply, without justification, that the estimates must be too high. Excess deaths during the pandemic are the difference between expected and actual deaths. In India, there are gaps in the data leading to uncertainties in both quantities. Taking into account uncertainties is a key part of analysing the data. The WHO team has stressed that their estimates are backed by such sensitivity analysis," says Banaji. 

Thus, according to him, there is no good reason to doubt the WHO estimates of mortality. "On the contrary, as each new study appears, it seems more and more absurd for the government to maintain that official COVID-19 deaths tell the whole story."

Dr Shahid Jameel, Fellow, OCIS and Green Templeton College, University of Oxford, argues, "The government has only criticised WHO methodology without coming up with its own methodology and estimates, despite an entire institute dedicated to epidemiology (NIE, Chennai; an ICMR institute). What the government should have done is to revise its own numbers beyond the official estimate that we know is not factual. Moreover, it continues to stonewall data available with Health Ministry by not making it available to independent researchers. Lack of transparency always breeds mistrust."

Giving a very basic calculation, Jameel points out that officially reported numbers are gross underestimates. 

"The crude death rate reported by India for 2018-19 (pre-COVID) was 7.2/1,000 people in a year. For a 1.38 billion population, this translates to about 27,200 deaths/day. The government estimates for COVID-19 deaths at the peak of the second wave were about 4,000/day, which is only about 15% more than normal daily deaths. This would not have caused the scenes we witnessed at crematoria and burial grounds."

Moreover, Jameel points out, "Many deaths that occurred in ICUs due to, say, respiratory failure were not counted as COVID deaths because prevailing ICMR guidelines did not count those as one if there was no accompanying positive COVID test result. The test reports of many never came. There must be lakhs of such people. This is one way of undercounting, not out of devious design, but due to the prevailing chaos."

Guilmoto argues that "official statistics often differ from indirect estimates or internationally recognised figures. In India, for instance, figures drawn from the CRS are often below true figures estimated from sample surveys or other indirect methods. This should be no cause for controversy because we owe the true death toll of the COVID pandemic to all those who suffered from this. The gap between figures should encourage decision-makers and statisticians to strengthen data collection and improve estimation methods."

Echoing Guilmoto's views, Banaji says, "The government "needs to acknowledge the scale of the tragedy. Lessons need to be learned about the management of epidemics and data and transparency."

The WHO report is expected to be published soon. After its release, at least now, for once, the government should stop contesting the estimated number of COVID-19 deaths. Instead, it should work with experts constructively to assess the damage done and ensure that we do not lose lives owing to our mistakes or being in denial.