Lancet paper on Chloroquine is overhyped - Real World Data should not be a black box

The use of Real World Evidence in Covid-19 is growing, and that’s a good thing. Properly implemented, RWE has the potential to deliver solid results faster than Randomized Controlled Trials, as I explained in a .css-mckguv{-webkit-transition:background 0.25s var(--ease-in-out-quad),color 0.25s var(--ease-in-out-quad);transition:background 0.25s var(--ease-in-out-quad),color 0.25s var(--ease-in-out-quad);color:var(--theme-ui-colors-accent,#6166DC);}.css-mckguv:visited{color:var(--theme-ui-colors-accent,#6166DC);opacity:0.85;}.css-mckguv:hover,.css-mckguv:focus{-webkit-text-decoration:underline;text-decoration:underline;}previous blog post.

So I couldn’t miss the largest observational study published to date on the effects of (hydroxy-)chloroquine, in 96 032 hospitalised Covid-19 patients, from an international registry comprising 671 hospitals in six continents:

Unfortunately, this paper is an example of Real World Evidence and Big Data implemented in the wrong way. Data is powerful, but not a magic bullet, and this kind of low-quality literature is my inspiration for launching the COVIND Consortium of Individual Patient Data.

In this Lancet piece, I identified 3 flaws:.css-1rpojxi{list-style:none;counter-reset:list;color:var(--theme-ui-colors-articleText,#08080B);position:relative;padding:15px 0 30px 30px;-webkit-transition:background 0.25s var(--ease-in-out-quad),color 0.25s var(--ease-in-out-quad);transition:background 0.25s var(--ease-in-out-quad),color 0.25s var(--ease-in-out-quad);margin:0 auto;font-size:18px;width:100%;max-width:680px;}@media (max-width:66.875em){.css-1rpojxi{max-width:507px;}}@media (max-width:45.9375em){.css-1rpojxi{max-width:486px;padding-left:0px;}}@media (max-width:33.75em){.css-1rpojxi{padding-left:20px;}}.css-1rpojxi li{position:relative;padding-bottom:15px;}@media (max-width:45.9375em){.css-1rpojxi li{padding-left:30px;}}@media (max-width:33.75em){.css-1rpojxi li{padding-left:30px;}}@media (max-width:45.9375em){.css-1rpojxi li p{padding:0;}}.css-1rpojxi li >:not(ol,ul){display:inline;}.css-1rpojxi li::before{width:3rem;display:inline-block;position:absolute;color:var(--theme-ui-colors-articleText,#08080B);}.css-1rpojxi li::before{content:'';position:absolute;left:-30px;top:8px;height:8px;width:8px;background:var(--theme-ui-colors-articleText,#08080B);}@media (max-width:45.9375em){.css-1rpojxi li::before{left:0;}}

No reproducibility

First, this study is not reproducible. Authors did not release their data, and it’s not even in their plans:

They argue legal pretexts, but it’s still a bad professional practice. Their data is a black box. In terms of reproducibility, this paper doesn’t meet community standards we have in machine learning, where most authors release their data and code on GitHub, and sometimes even build a ready-made container for one-click reproducibility on Google AI Hub, or other cloud platforms.

As expected, this data secrecy implies some weird stuff. For example, some critics, like in this blog from Columbia University, noted more deaths in this Lancet paper than reported by Australian authorities. Are Australian authorities hiding something, or did Lancet authors fabricated data for their cherished conclusion? I trust Australian authorities more.

A lot of other issues spill around in this crazy dataset, here and elsewhere. Garbage in, garbage out.

No data traceability

This Lancet paper also has no data traceability. Authors didn’t release the names of people who were responsible for data collection at hospitals. That’s the second black box. The paper only has 4 co-authors (ridiculously small for medical literature standards), and only one person was responsible for data acquisition: Sapan S Desai. How could he curate a huge dataset of 96 032 patients alone, so quickly? Is his company Surgisphere magical? Surgisphere only has 6 employees on LinkedIn. That’s insane.

Surgisphere claims to be “a leader in healthcare data analytics”, but their LinkedIn posts only have 1 like (a self-like!), and their oldest post is 2-months old. This bold and shady company seems a bit early-stage for leadership claims. It’s not the Google of health data.

Why didn’t doctors and their labs try to stick their names on this prestigious Lancet paper? So many grants and tenures missed.

As a comparison, in particle physics, the important paper discovering the Higgs Boson particle at the Large Hadron Collider has 5,154 co-authors.

At COVIND data consortium, we are building a network of local doctors, who are also scientific partners. They put their names on all papers where their data is involved. They often tell me that’s their first condition for collaborating. That is not only to acknowledge their scholarly contribution. Their signature is also a public endorsement of the dataset they release, and a key link in the data traceability chain. That’s important when raw identified data is inaccessible for privacy reasons.

Of course, mistakes can happen, we are moving fast, but we implemented transparent error-correction procedures.

No review transparency

Lancet did not release the names of the referees (if any). Yet another black box. So the only signed editorial validation comes from Lancet editor-in-chief Richard Horton.

Richard Horton has an old political conflict with the US President. Donald Trump happens to be a big fan of Chloroquine, and Trump doesn’t care about Randomized Clinical Trials, and about science in general. Horton is certainly tempted to prove him wrong. In this bestial battle of egos, did Horton feel so insecure as to let a fraudulent submission slip through editorial filters? Oups!

Richard Horton published several political rants against Trump in The Lancet:

I don’t trust Fox News about global warming, so why should I trust The Lancet about Trump-approved Chloroquine? Fake news all around.

As written elsewhere in The Lancet (I love it as a political outlet, I am left-wing too!): “public health should not be guided by partisan politics”

As a left-wing citizen, I agree with Richard Horton when he says that this pandemic is “the biggest science policy failure in a generation”, but I am surprised he wanted to be a part of it. So ironic.

From the ashes of the Lancet, I am still hopeful that a new era for data-driven medicine will emerge, based on three pillars: reproducibility, traceability and transparency.

And there is an open letter to the Lancet authors.

READ THE OPEN LETTER HERE

In an open letter to the study authors and the editor of the Lancet a number of concerns are raised.

1. The numbers in the article don't compare to official numbers
2. The research group is, so far, not sharing any of their data or hospitals. The Lancet is a signatory to an international consortium of research groups and publishers that any and all COVID data that is published has to be open source.
3. Hospitals that had to have been included are saying they were never contacted or gave permission for their information to be shared.

I think it's important to point out that when trying to determine the veracity of scientific data, the underlying motivations of those that are posting/publishing must be considered. The signatories on this letter include researchers who are on record as being highly skeptical of HCQ for COVID. When any person or group doubles down on something "scientific" that a large group of people are questioning, that is a big red flag (in medical research at least) that there are motivations other than the truth. Which is why all these conspiracy theories can be debunked so quickly. Most true scientists welcome others attempting to replicate, criticize, and tear apart their work so that the truth can be had. So far, the study authors' only public responses have been defensive. This, combined with some of the people who signed the letter, make me worry that there is more going on here and, if the group does not completely release all of their data publicly, I wouldn't be surprised to see the Lancet issue a retraction.

Here is the text of the letter

Open letter to MR Mehra, SS Desai, F Ruschitzka, and AN Patel, authors of“Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis”. Lancet. 2020 May 22:S0140-6736(20)31180-6. doi: 10.1016/S0140-6736(20)31180-6. PMID: 32450107and to Richard Horton (editor of The Lancet).

Concerns regarding the statistical analysis and data integrity

The retrospective, observational study of 96,032 hospitalized COVID-19 patients from six continents reported substantially increased mortality (~30% excess deaths) and occurrence of cardiac arrhythmias associated with the use of the 4-aminoquinoline drugs hydroxychloroquine and chloroquine. These results have had a considerable impact on public health practice and research. The WHO has paused recruitment to the hydroxychloroquine arm in their SOLIDARITY trial.The UK regulatory body, MHRA,requested the temporary pausing of recruitment into all hydroxychloroquine trials in the UK (treatment and prevention), and France has changed its national recommendation for the use of hydroxychloroquine in COVID-19 treatment and also halted trials. The subsequent media headlines have caused considerable concern to participants and patients enrolled in randomized controlled trials (RCTs) seeking to characterize the potential benefits and risks of these drugs in the treatment and prevention of COVID-19 infections.

There is uniform agreement that well conducted RCTs are needed to inform policies and practices. This impact has led many researchers around the world to scrutinize in detail the publication in question. This scrutiny has raised both methodological and data integrity concerns. The main concerns are listed as follows:

1.There was inadequate adjustment for known and measured confounders (disease severity, temporal effects, site effects, dose used).

2.The authors have not adhered to standard practices in the machine learning and statistics community. They have not released their code or data. There is no data/code sharing and availability statement in the paper. The Lancet was among the many signatories on the Wellcome statement on data sharing for COVID-19 studies.

3.There was no ethics review.

4.There was no mention of the countries or hospitals that contributed to the data source no acknowledgments to their contributions. A request to the authors for information on the contributing centres was denied.

5.Data from Australia are not compatible with government reports (too many cases for just five hospitals, more in-hospital deaths than had occurred in the entire country during the study period). Surgisphere (the data company) have since stated this was an error of classification one hospital from Asia. This indicates the need for further error checking throughout the database.

6.Data from Africa indicate that nearly 25% of all COVID-19 cases and 40% of all deaths in the continent occurred in Surgisphere-associated hospitals which had sophisticated electronic patient data recording, and patient monitoring able to detect and record “nonsustained [at least 6 secs] or sustained ventricular tachycardia or ventricular fibrillation”. Both the numbers of cases and deaths, and the detailed data collection, seem unlikely.

7.Unusually small reported variances in baseline variables, interventions and outcomes between continents(Table S3).

8.Mean daily doses of hydroxychloroquine that are 100 mg higher than FDA recommendations, whereas 66% of the data are from North American hospitals.9.Implausible ratios of chloroquine to hydroxychloroquine use in some continents.10.The tight 95% confidence intervals reported for the hazard ratios appear inconsistent with the data. For instance,for the Australian Data this would need about double the numbers of recorded deaths were reported in the paper.The patient data were obtained through electronic health records, supply chain databases,and financial records. The data are held by the US company Surgisphere. In response to a request for the data Professor Mehra replied:“Our data sharing agreements with the various governments, countries and hospitals do not allow us to share data unfortunately.” Given the enormous importance and influence of these results, we believe it is imperative that:

1.The company Surgisphere Provides details on data provenance. At the very minimum, this means sharing the aggregated patient data at the hospital level (for all covariates and outcomes)

2.Independent validation of the analysis is performed by a group convened by the World Health Organization,or at least one other independent and respected institution. This would entail additional analyses (e.g. determining if there is a dose-effect)to assess the validity of the conclusions

3.There is open access to all the data sharing agreements cited above to ensure that,in each jurisdiction,any mined data was legally and ethically collected and patient privacy aspects respected

In the interests of transparency, we also ask The Lancet to make openly available the peer review comments that led to this manuscript to be accepted for publication.This open letter is signed by clinicians, medical researchers, statisticians, and ethicists from across the world. The full list of signatories and affiliations can be found below.

Click here for link at https://zenodo.org/record/3865253#.XtEi_jpKjZs

HERE IS THE FULL LIST OF SIGNATORIES TO THE LETTER ( I think that’s what you are interested in ):

SOURCE: https://www.facebook.com/avalonwebagency/posts/2948493341900632

This open letter is signed by clinicians, medical researchers, statisticians, and ethicists from across the world.

TITLE: Open letter to The Lancet :

“Your study is irregular and in a blackbox. Transparency is mandatory by the scientific-publication-codes you have signed, and it is needed to verify all the discrepancies of this study.”

LIST OF SIGNATORIES :

Dr James Watson (Statistician, Mahidol Oxford Tropical Medicine Research Unit, Thailand)1
Professor Amanda Adler (Trialist & Clinician, Director of the Diabetes Trials Unit, UK)
Dr Ravi Amaravadi (Researcher, University of Pennsylvania, USA)
Dr Ambrose Agweyu (Medical researcher, KEMRI-Wellcome Trust Research Programme, Kenya) Professor Michael Avidan (Clinician, Washington University in St Louis, USA)
Professor Nicholas Anstey (Clinician, Menzies School of Health Research, Australia)
Dr Yaseen Arabi (Clinician, King Saud Bin Abdulaziz University for Health Sciences, Saudi Arabia)
Dr Elizabeth Ashley (Clinician, Director of the Lao-Oxford-Mahosot Hospital-Wellcome Trust Research Unit, Laos) Professor Kevin Baird (Researcher, Head of the Eijkman-Oxford Clinical Research Unit, Indonesia)
Professor Francois Balloux (Researcher, Director of the UCL Genetics Institute, UK)
Dr Clifford George Banda (Clinician, University of Cape Town, South Africa)
Dr Edwine Barasa (Health economist, KEMRI-Wellcome Trust Research Programme, Kenya)
Professor Karen Barnes (Clinical Pharmacology, University of Cape Town, South Africa)
Professor David Boulware (Researcher & Triallist, University of Minnesota, USA)
Professor Buddha Basnyat (Clinician, Head of the Oxford University Clinical Research Unit - Nepal, Nepal) Professor Philip Bejon (Medical researcher, Director of the KEMRI-Wellcome Trust Research Programme, Kenya) Professor Mohammad Asim Beg (Clinician/Researcher, Aga Khan University, Pakistan)
Professor Emmanuel Bottieau (Clinician, Institute of Tropical Medicine, Antwerp, Belgium)
Dr Sabine Braat (Statistician, University of Melbourne, Australia)
Professor Frank Brunkhorst (Clinician, Jena University Hospital, Germany)
Dr Todd Campbell Lee (Researcher, McGill University, Canada)
Professor Caroline Buckee (Epidemiologist, Harvard TH Chan School of Public Health, USA)
Dr James Callery (Clinician, Mahidol Oxford Tropical Medicine Research Unit, Thailand)
Professor John Carlin (Statistician, University of Melbourne & Murdoch Children’s Research Institute, Australia) Dr Nomathemba Chandiwana (Research Clinician, University of the Witwatersrand, South Africa)
Dr Arjun Chandna (Clinician, Cambodia Oxford Medical Research Unit, Cambodia)
Professor Phaik Yeong Cheah (Ethicist/Pharmacist, Mahidol Oxford Tropical Medicine Research Unit, Thailand) Professor Allen Cheng (Clinician, Monash University, Australia)
Professor Leonid Churilov (Statistician, University of Melbourne, Australia)
Professor Ben Cooper (Epidemiologist, University of Oxford, UK)
Dr Cintia Cruz (Paediatrician Mahidol Oxford Tropical Medicine Research Unit, Thailand)
Professor Bart Currie (Director, HOT NORTH, Menzies School of Health Research, Australia)
Professor Joshua Davis (Clinician, President of the Australasian Society for Infectious Diseases, Australia)
Dr Jeremy Day (Clinician, Oxford University Clinical Research Unit, Vietnam)
Professor Nicholas Day (Clinician, Director of the Mahidol Oxford Tropical Medicine Research Unit, Thailand)
Dr Hakim-Moulay Dehbi (Statistician, University College London, UK)
Dr Justin Denholm (Clinician, Researcher, Ethicist, Doherty Institute, Australia)
Dr Lennie Derde (Intensivist/Researcher, University Medical Center Utrecht, The Netherlands)
Professor Keertan Dheda (Clinician/Researcher, University of Cape Town, & Groote Schuur Hospital, South Africa) Dr Mehul Dhorda (Clinical Researcher, Mahidol Oxford Tropical Medicine Research Unit, Thailand)
Professor Annane Djillali (Dean of the School of Medicine, Simone Veil Université, France)
Professor Arjen Dondorp (Clinician, Mahidol Oxford Tropical Medicine Research Unit, Thailand)
Dr Joseph Doyle (Clinician, Monash University and Burnet Institute, Australia)
Dr Anthony Etyang (Medical Researcher, KEMRI-Wellcome Trust Research Programme, Kenya)
Dr Caterina Fanello (Epidemiologist, University of Oxford, UK)
Professor Neil Ferguson (Epidemiologist, Imperial College London, UK)
Professor Andrew Forbes (Statistician, Monash University, Melbourne, Australia)
Professor Oumar Gaye (Clinical Researcher, University Cheikh Anta Diop, Senegal)
Dr Ronald Geskus (Head of Statistics at the Oxford University Clinical Research Unit, Vietnam)
Professor Dave Glidden (Biostatistics, University of California, USA)
Professor Azra Ghani (Epidemiologist, Imperial College London, UK)
Prof Philippe Guerin (Medical researcher, University of Oxford, UK)
Dr. Raph Hamers (Clinician/Trialist, Eijkman-Oxford Clinical Research Unit, Indonesia)
Professor Peter Horby (Clinical Researcher, Centre for Tropical Medicine and Global Health, University of Oxford) Dr Jens-Ulrik Jensen (Clinician/Trialist, University of Copenhagen, Denmark)
Dr Hilary Johnstone (Clinical Research Physician, Independent)
Professor Kevin Kain (Clinical Researcher, University of Toronto, Canada)
Dr Sharon Kaur (Ethicist, University of Malaya, Malaysia)
1 For correspondence: james@tropmedres.ac
Dr Evelyne Kestelyn (Head of Clinical Trials, Oxford University Clinical Research Unit, Vietnam)
Dr Tan Le Van (Medical Researcher, Oxford University Clinical Research Unit, Vietnam)
Professor Katherine Lee (Statistician, University of Melbourne, Australia)
Professor Laurence Lovat (Clinical Director of Wellcome EPSRC Centre for Interventional & Surgical Sciences, UCL, UK) Professor Kathryn Maitland (Clinician, Imperial College London/KEMRI Wellcome Trust Programme, Kenya)
Dr Julie Marsh (Statistician, Telethon Kids Institute, Australia)
Professor John Marshall (Clinician/Researcher, University of Toronto, Canada)
Dr Gary Maartens (Clinician, University of Cape Town, South Africa)
Professor Mayfong Mayxay (Clinician/Researcher, Lao-Oxford-Mahosot Hospital-Wellcome Trust Research Unit, Laos) Dr John McKinnon (Clinician/Researcher, Wayne State University, USA)
Dr Laura Merson (Clinical researcher, University of Oxford, UK)
Dr Alistair McLean (Medical researcher, University of Oxford, UK)
Professor Ramani Moonesinghe (Clinician researcher, University College London, UK)
Professor Bryan McVerry (Medical researcher, University of Pittsburgh, USA)
Professor William Meurer (Clinician/Medical researcher, University of Michigan, USA)
Dr Kerryn Moore (Epidemiologist, London School of Hygiene and Tropical Medicine, UK)
Dr Rephaim Mpofu (Clinician, University of Cape Town, South Africa)
Dr Mavuto Mukaka (Statistician, Mahidol Oxford Tropical Medicine Research Unit, Thailand)
Dr Srinivas Murthy (Clinical Researcher, University of British Columbia, Canada)
Professor Kim Mulholland (Clinician, London School of Hygiene & Tropical Medicine, UK)
Professor Alistair Nichol (Clinician Researcher, Monash University, Australia)
Professor Francois Nosten (Clinician, Director of the Shoklo Malaria Research Unit, Thailand)
Dr Matthew O’Sullivan (Clinician, Westmead Hospital & University of Sydney, Australia)
Professor Piero Olliaro (Clinical Researcher, University of Oxford, UK)
Professor Marie Onyamboko (Clinical researcher, Kinshasa School of Public Health, DRC)
Dr Marcin Osuchowski (Medical researcher, Ludwig Boltzmann Institute, Austria)
Professor Catherine Orrell (Clinical Pharmacologist, University of Cape Town, South Africa)
Professor Jean Bosco Ouedraogo (Medical Researcher, WWARN, Burkina Faso)
Dr Elaine Pascoe (Statistician, University of Queensland, Australia)
Professor David Paterson (Clinician, Director, UQ Centre for Clinical Research, Australia)
Dr Kajaal Patel (Paediatrician, Cambodia Oxford Medical Research Unit, Cambodia)
Dr Tom Parke (Statistician, Berry Consultants, UK)
Professor Philippe Parola (Researcher, Aix-Marseille University, France)
Professor Paul Newton (Clinician, University Oxford, UK)
Professor David Price (Statistician, Doherty Institute & University of Melbourne, Australia)
Professor Richard Price (Clinician, Menzies School of Health Research, Australia)
Professor Sasithon Pukrittayakamee (Clinician, Mahidol University, Thailand)
Dr Ben Saville (Statistician, Berry Consultants & Vanderbilt University)
Professor Jason Roberts (Pharmacist/Clinician, The University of Queensland, Australia)
Professor Stephen Rogerson (Clinician, University of Melbourne, Australia)
Professor Kathy Rowan (Researcher, Director of the ICNARC Clinical Trials Unit, UK)
Dr William Schilling (Clinician, Mahidol Oxford Tropical Medicine Research Unit, Thailand)
Dr Anuraj Shankar (Clinician/Trialist, Eijkman-Oxford Clinical Research Unit, Indonesia)
Professor Sanjib Kumar Sharma (Clinician, Koirala Institute of Health Sciences, Nepal)
Professor Julie Simpson (Statistician, University of Melbourne, Australia)
Professor Frank Smithuis (Clinical researcher, Director of the Myanmar Oxford Tropical Research Unit, Myanmar)
Dr Tim Spelman (Statistician, Burnet Institute, Australia)
Dr Kasia Stepniewska (Statistician, University of Oxford, UK)
Dr Nathalie Strub Wourgaft (Clinician, Drugs for Neglected Diseases initiative, Switzerland)
Dr Aimee Taylor (Statistician, Harvard T.H. Chan School of Public Health, USA)
Dr Walter Taylor (Clinician, Mahidol Oxford Tropical Medicine Research Unit, Thailand)
Professor Guy Thwaites (Clinician, Director of the Oxford University Clinical Research Unit, Vietnam)
Professor Tran Tinh Hien (Clinician, Oxford Clinical Research Unit, Vietnam)
Professor Steven Tong (Clinician, University of Melbourne, Australia)
Professor Paul Turner (Clinician/Researcher, Director of Cambodia Oxford Medical Research Unit, Cambodia) Professor Ross Upshur (Head of Division of Clinical Public Health, University of Toronto, Canada)
Professor Rogier van Doorn (Clinical Microbiologist, University of Oxford, UK)
Professor Sir Nicholas White (Clinician, Mahidol Oxford Tropical Medicine Research Unit, Thailand)
Professor Thomas Williams (Clinician, KEMRI-Wellcome Trust Research Programme, Kenya)
Professor Chris Woods (Researcher, Duke University, USA)
Dr Sophie Yacoub (Clinician, Oxford University Clinical Research Unit, Vietnam)
Professor Marcus Zervos (Researcher, Wayne State University School of Medicine, USA)

Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.