Natural experiments in medicine

From Ganfyd

Jump to: navigation, search

Sometimes experiments are so difficult to justify ethically by the evidence base that only governmental bodies, fringe and special interest groups and the Press are likely to promote and enable them. When one of these natural experiments occurs we can observe and learn from it. Sometimes a major advance in understanding occurs, more often the evidence base turns out to be right. Sometimes the controversy simply takes a slightly different tack when revisited. This page is dedicated to those willing to learn from history.

Serendipity from coincidences between diseases and events is also worth noting. One of the early drugs tried against tuberculosis was not very good, but patients in the arm of the trial receiving it, were noted to be less miserable or depressed. It went into use as an early anti-depressant. An interesting substance with the potential to reduce blood pressure by its dilating effect on arteries was tried for this indication. It turned out to be disappointing, although it is now important in pulmonary hypertension. However sildenafil proved very useful for one of its unanticipated effects, on ability to sustain erections and the class of drugs that results continues to be useful in the indication of sexual impotence as well as being seen as a lifestyle drug.



Let nature take its course experiments

In Boston around 1721 there was argument over variolation with churchmen on both sides "...within a year or two after the first experiment nearly three hundred persons had been inoculated by Boylston in Boston and neighbouring towns, and out of these only six had died; whereas, during the same period, out of nearly six thousand persons who had taken smallpox naturally, and had received only the usual medical treatment, nearly one thousand had died."[1]

Natural experiment around Boston
Total DiedMortality %
Variolated C 300 6C 2%
UnvariolatedC 6000C 1000C 14%*

gives it as about 14% rather than 17%, likely with closer figures.

Drug regulation experiments

Age specific mortality in males from ischaemic heart disease in Australia (blue) and USA (red) with introduction propranolol indicated respectively. The peak mortality occurred in 1965 in Australia and 1970 in the USA. Any possible correlation would be confounded by factors such as change in smoking habit and other drug introductions. In England and Wales male mortality from IHD, started falling consistently from 1965, a year after propranolol was first marketed. See (Report from Nat Statistical Office)

The US FDA delayed acceptance of the first beta-blocker, propranolol, for a few years due to unjustified safety concerns as it transpired. Indeed the USA has demonstrated repeatedly the problems of regulatory balance due to the impact of political initiatives on comparative public health measures. Prohibition of alcohol did not solve the problem although it had direct public heath benefits such as initially a marked fall in cirrhosis, normalisation to domestic consumption with reduction in per capita consumption that took almost 40 years to reach pre-prohibition levels[2] and reduced issues such as foetal alcohol syndrome incidence that was totally unrecognised at the time. Indeed prohibition played a role in the delayed recognition in the medical community of the significance of chronic and late onset diseases associated with alcohol intake as an issue[3]. Strict controlled drug regulation created other problems in due course so many of the more extreme natural experiments that fuel the controlled drugs and public health debate come from the USA or the influence of its policy on other nations. Poor use of heavily promoted narcotics by the medical profession for use in chronic pain is likely to have been a factor in causing an epidemic of overdose deaths in the USA[4].

Anti-vaccinationists experiments

  • Toward the end of the 19th century in the USA smallpox outbreaks had become contained. Around that time anti-vaccinationists became more active, asserting the line of William Tebb that smallpox had diminished because of improved diet and health of the population, that the disease was less virulent and offered little risk to people who caught it young, and that vaccination in fact caused smallpox. Vaccination fell into disuse. However, in the 1870s the disease became epidemic in what was now clearly a susceptible population. As states attempted to enforce existing vaccination laws or pass new ones, vigorous anti-vaccination movements arose. Nevertheless, vaccination increased, and yet again, Smallpox decreased and is now absent.
  • In the 1970's, the Sunday Times and various anti-vaccinationists managed to produce a fall in childhood immunisations. This was followed by an increase in the diseases[2], with accompanying deaths and injuries. And then repeated with the mixed Measles Mumps and Rubella immunisation.
  • In Nigeria, and therefore around Nigeria shortly afterward, in 2003, a political movement against polio vaccine greatly reduced its administration. Polio cases and the area they had been found in had been shrinking steadily, but now they began to expand.

See also [3].

  • In Japan, MMR vaccine was introduced in 1989, but the programme was terminated in 1993 and only single vaccines used thereafter Bandolier. "The incidence of all autistic spectrum disorders, and of autism, continued to rise after MMR vaccine was discontinued. The incidence of autism was higher in children born after 1992 who were not vaccinated with MMR than in children born before 1992 who were vaccinated. The incidence of autism associated with regression was the same during the use of MMR and after it was discontinued."[4] The conclusion reached by Bandolier on this elegant and well-observed natural experiment with 278 cases was that this destroys any possible causative link between use of the vaccine and autism. Whatever causes autism, it is not the MMR vaccine.[5]

Designer drugs experiments

In 1982 there was an epidemic of MPTP-induced parkinsonism in California repeating and confirming the 1976 case report suggesting that making a mistake in the synthesis of MPPP (1-Methyl-4-phenyl-4-piperidinol propionate), a synthetic opioid, produces MPTP which has been known since 1947 as a fairly selective neurotoxin to the substantia nigra. The neurotoxin theory of Parkinson's disease is offered as partial explanation for these and several other neurodegenerative conditions.

Statins not all equivalent experiments

In New Zealand in 1996, fluvastatin became the reference price statin. This motivated patients to take the "free" drug, at inadequate doses, and a neat bit of epidemiology by Professor Jim Mann in Otago showed a not unexpected increase in vascular thrombotic events.[5].

Weekend morbidity and mortality experiments

Differental mortality for four year cohorts of acute myocardial infarction in New Jersey, showing deteriorating survival in the 1999-2002 cohort compared to the previous 8 years and differential survival for every 4 year period in favour of weekday admissions rather than weekend (prefix WE) admissions[6]
  • Whole population mortality is quite different from institutional mortality and both can be influenced by ease of access to health and social care. This in turn has both patient components and care giver components which in many societies appears likely to vary during the week due to employment and other cultural patterns.
  • The contribution of illness severity and admission time has been explored in Western healthcare systems and was better clarified by big data analysis of blood pathology results. This showed that for emergency admissions time of presentation during the working week and weekend presentation was related to illness severity. Severe illness in common presentations reaches UK hospitals between 11:00 h and 15:00 h[7]. Severity seems to predict outcome on admissions on public holidays better than on Saturdays or Sundays.It is known that only some illness presentations, usually those with non infectious or non obvious trauma emergency component, are associated with a time of week effect[8]. For example pulmonary embolism that can be difficult to diagnose has a strong weekend effect while meningococcal disease does not.
  • Hospital workload is not associated with mortality in a "stabilised" public healthcare system[7]. Morbidity (but not mortality) in private systems is linked to out of hours presentations of complications[9].
  • For religious, social and political reasons we have the concept of the working week and weekends. Cultures make varying provision for emergency health care at weekends but usually limit access to certain healthcare resources at weekends. This has been associated with differential mortality in patients with common medical illnesses depending upon what day they are admitted to hospital creating the 'weekend effect' on mortality. The most convincing evidence for such an effect comes from whole population studies. [10][11][12][13][14][15][16][17]. One issue that has to be controlled for, is that patients admitted at weekends are more ill[18]. A common theme in commentary on this issue is that resources necessary to manage the patients are not available at weekends due to logistic reasons. To a degree this can be addressed by systems changes and there is likely to be an optimum resource distribution over time during a week for any health and care system treating a particular condition. Primary care and social care constraints could create an exaggerated weekend effect in secondary care if secondary care resources are used at weekends instead of more effective primary and social care resources. Such patterns have been suggested to be important and need to be considered.
  • There is evidence from the UK that the shift in emergency surgical orientated resources and practice in the last 15 years that has resulted in the 30 day mortality rate for emergency surgical admissions falling from 5·4% (2000-2004) to 2·9 % during 2010-2014 has largely abolished the weekend effect in this speciality[19]. This reduction happened before a recent overt political drive to address the issue. The decrease in weekend mortality with evolution of other health systems over time has also been well described[20]. In the Netherlands hip fracture excess mortality at weekends has been successfully designed out of the system[21].
  • With the ever more acutely orientated interventions to treat myocardial infarction this is not unexpectedly the case with this diagnosis. It has been estimated that 9 to 10 deaths per 1000 admissions could be saved by better acute service cover. However some evidence, while it shows ever increasing acute survival, also shows worsening long term survival.[22] An explanation for this is that many recently introduced interventions in acute myocardial infarction are based on short term studies and can have high take up because of the assumption that short term effectiveness translates into long term effectiveness. Modern acute interventions as practised in the Eastern seaboard of the United States while reducing immediate mortality in myocardial infarction patients who make it to hospital could be increasing medium term mortality by up to 30 deaths per 1000 patients treated with part of the issue being time of presentation or relapse. The data is being watched very carefully. By 2016 this fear appeared to be justified with respect to access to PCI being reduced at weekends[23][24]. Accordingly for illnesses that require prompt high technology, highly trained specialist care for optimal outcome it seems likely that the weekend effect reflects lack of appropriate resources. However much emergency care mortality at weekends relates to illnesses such as infection and trauma where the technology and specialist care part of the pathway of care is apparently available.
  • The hyper actuate stroke model introduced during the last two decades worldwide has created major resourcing and systems challenges but also more detailed data capture possibilities. Detailed analysis for stroke in the UK reveal that the mortality effect is more a time of day effect, with for example 30 day mortality showing no weekend effect but there is an out of hours effect at night most pronounced during week days[25]. The differential access to investigations has been long known in stroke[26]. However the registries created to define stroke care have also allowed us to understand just how great a factor coding error is in creating the 'weekend effect' from routine admissions data[27].
  • It has been known for many years that seasonal influenza confers the greatest increase in risk of in-hospital mortality, followed by weekend admission, high hospital bed occupancy and that increasing nurse staffing levels decreases the absolute risk of mortality as the most effective potential intervention.[28]. However this translates into resource constraints that are not attractive political targets, as for example increasing hospital bed capacity which requires capital funding, tends to result in unmeet demand and diversion from other pathways of care filling the beds. Paying for nurses and their training is the single largest recurrent funding component in healthcare.
  • Therefore the phenomenon has been attributed in some healthcare systems to reduced availability of senior clinical staff and reduced access to investigative services in hospitals at weekends[29] but on a whole system basis there is no causal evidence establishing this link[30]. However from 2015 in England experiments in redeploying secondary care medical manpower were launched in terms of contract change justified by a government manifesto commitment to reduce weekend deaths. As it transpired higher mortality rates amongst emergency patients admitted to hospital at weekends reflected a lower probability of admission [31] partially driven by lack of GP services at weekends and GP services had a manpower crisis due to lack of investment so could not address the issue, the impact of other change to produce the same ends is unknown. There are actually multiple complexities as data modelling trying to control for case mix always has limitations, association is not causation, and the issues causing the phenomena are much wider than healthcare. As of 2016 there is actually no statistical evidence in the UK that the lower availability of specialised senior staff at weekends impacts mortality or that increasing this ratio alone as an intervention as commenced in 2014 decreases mortality[32]. No studies suggest junior doctor availability is more important than skilled nurses availability but this former issue can be seen to be easier to address in the context that shortages of both resources appear to be associated with poorer patient outcome and there is lower turnover of the doctor resource.

Medical training experiments

Modernising Medical Careers was introduced in the UK from 2007. The UK big bang approach to major alterations to the training of doctors (future GPs as well as the hospital specialists it is ostensibly aimed at) qualifies as a natural experiment. The results will take some time to become clear, indeed the architects of it may discover those results at first hand if they suffer ill health after retirement. Whatever the experiment has had several major modifications such as those resulting from the 2008 Tooke report[33] and the 2013 report[34]. By 2016 it had almost certainly become a factor in why junior doctors remaining in England were willing to withdraw their labour completely during industrial action.


  2. Blocker JS. Did prohibition really work? Alcohol prohibition as a public health innovation. American journal of public health. 2006 Feb; 96(2):233-43.(Link to article – subscription may be required.)
  3. Katcher BS. The post-repeal eclipse in knowledge about the harmful effects of alcohol. Addiction (Abingdon, England). 1993 Jun; 88(6):729-44.
  4. Volkow ND, McLellan AT. Opioid Abuse in Chronic Pain--Misconceptions and Mitigation Strategies. The New England journal of medicine. 2016 Mar 31; 374(13):1253-63.(Link to article – subscription may be required.)
  5. Begg E, Sidwell A, Gardiner S, Nicholls G, Scott R. The sorry saga of the statins in New Zealand--pharmacopolitics versus patient care. N Z Med J. 2003 Mar 14;116(1170) subscription needed to link
  6. Kostis WJ, Demissie K, Marcella SW, Shao YH, Wilson AC, Moreyra AE; Myocardial Infarction Data Acquisition System (MIDAS 10) Study Group. Weekend versus weekday admission and mortality from myocardial infarction. N Engl J Med. 2007;356(11):1099-109.
  7. a b Walker AS, Mason A, Quan TP, Fawcett NJ, Watkinson P, Llewelyn M, Stoesser N, Finney J, Davies J, Wyllie DH, Crook DW, Peto TEA. Mortality risks associated with emergency admissions during weekends and public holidays: an analysis of electronic health records. Lancet (London, England). 2017 May.(Print-Electronic) (Link to article – subscription may be required.)
  8. Hoshijima H, Takeuchi R, Mihara T, Kuratani N, Mizuta K, Wajima Z, Masaki E, Shiga T. Weekend versus weekday admission and short-term mortality: A meta-analysis of 88 cohort studies including 56,934,649 participants. Medicine. 2017 Apr; 96(17):e6685.(Print) (Link to article – subscription may be required.)
  9. Morgan DJR, Ho KM, Ong YJ, Kolybaba ML. Out-of-office hours' elective surgical intensive care admissions and their associated complications. ANZ journal of surgery. 2017 Jun.(Print-Electronic) (Link to article – subscription may be required.)
  10. Bell CM, Redelmeier DA. Mortality among patients admitted to hospitals on weekends as compared with weekdays.N Engl J Med. 2001;345(9):663-8. Erratum in:N Engl J Med 2001;345(21):1580.
  11. Saposnik G, Baibergenova A, Bayer N, Hachinski V. Weekends: A Dangerous Time for Having a Stroke? Stroke. 2007 Mar 8;
  12. Barba R, Losa JE, Velasco M, Guijarro C, Garcia de Casasola G, Zapatero A. Mortality among adult patients admitted to the hospital on weekends. Eur J Intern Med. 2006 Aug;17(5):322-324.
  13. Foss NB, Kehlet H. Short-term mortality in hip fracture patients admitted during weekends and holidays. British journal of anaesthesia 2006;96:450-4. (Direct link – subscription may be required.)
  14. Aylin P, Yunus A, Bottle A, Majeed A, Bell D. Weekend mortality for emergency admissions. A large, multicentre study. Quality & safety in health care. 2010 Jun; 19(3):213-7.(Link to article – subscription may be required.)
  15. Aylin P, Alexandrescu R, Jen MH, Mayer EK, Bottle A. Day of week of procedure and 30 day mortality for elective surgery: retrospective analysis of hospital episode statistics. BMJ (Clinical research ed.). 2013; 346:f2424.(Epub)
  16. Freemantle N, Richardson M, Wood J, Ray D, Khosla S, Shahian D, Roche WR, Stephens I, Keogh B, Pagano D. Weekend hospitalization and additional risk of death: an analysis of inpatient data. Journal of the Royal Society of Medicine. 2012 Feb; 105(2):74-84.(Link to article – subscription may be required.)
  17. Ruiz M, Bottle A, Aylin PP. The Global Comparators project: international comparison of 30-day in-hospital mortality by day of the week. BMJ quality & safety. 2015 Aug; 24(8):492-504.(Link to article – subscription may be required.)
  18. Mohammed MA, Faisal M, Richardson D, Howes R, Beaston K, Speed K, Wright J. Adjusting for illness severity shows there is no difference in patient mortality at weekends or weekdays for emergency medical admissions. QJM : monthly journal of the Association of Physicians. 2016 Jul 12.(Epub ahead of print) (Link to article – subscription may be required.)
  19. McCallum IJ, McLean RC, Dixon S, O'Loughlin P. Retrospective analysis of 30-day mortality for emergency general surgery admissions evaluating the weekend effect. The British journal of surgery. 2016 Aug 12.(Epub ahead of print) (Link to article – subscription may be required.)
  20. Conway R, Cournane S, Byrne D, O'Riordan D, Silke B. Improved mortality outcomes over time for weekend emergency medical admissions. Irish journal of medical science. 2017 May.(Print-Electronic) (Link to article – subscription may be required.)
  21. Nijland LMG, Karres J, Simons AE, Ultee JM, Kerkhoffs GMMJ, Vrouenraets BC. The weekend effect for hip fracture surgery. Injury. 2017 Jul; 48(7):1536-1541.(Print-Electronic) (Link to article – subscription may be required.)
  22. Kostis WJ, Demissie K, Marcella SW, Shao YH, Wilson AC, Moreyra AE; Myocardial Infarction Data Acquisition System (MIDAS 10) Study Group. Weekend versus weekday admission and mortality from myocardial infarction. N Engl J Med. 2007;356(11):1099-109.
  23. Khoshchehreh M, Groves EM, Tehrani D, Amin A, Patel PM, Malik S. Changes in mortality on weekend versus weekday admissions for Acute Coronary Syndrome in the United States over the past decade. International journal of cardiology. 2016 May 1; 210:164-72.(Link to article – subscription may be required.)
  24. Kumar G, Deshmukh A, Sakhuja A, Taneja A, Kumar N, Jacobs E, Nanchal R. Acute myocardial infarction: a national analysis of the weekend effect over time. Journal of the American College of Cardiology. 2015 Jan 20; 65(2):217-8.(Link to article – subscription may be required.)
  25. Bray BD, Cloud GC, James MA, Hemingway H, Paley L, Stewart K, Tyrrell PJ, Wolfe CDA, Rudd AG. Weekly variation in health-care quality by day and time of admission: a nationwide, registry-based, prospective cohort study of acute stroke care Lancet 10 May 2016 DOI:
  26. Palmer WL, Bottle A, Davie C, Vincent CA, Aylin P. Dying for the weekend: a retrospective cohort study on the association between day of hospital presentation and the quality and safety of stroke care. Archives of neurology. 2012 Oct; 69(10):1296-302.(Link to article – subscription may be required.)
  27. Li L, Rothwell PM. Biases in detection of apparent "weekend effect" on outcome with administrative coding data: population based study of stroke. BMJ (Clinical research ed.). 2016; 353:i2648.(Epub)
  28. Schilling PL, Campbell DA, Englesbe MJ, Davis MM. A comparison of in-hospital mortality risk conferred by high hospital occupancy, differences in nurse staffing levels, weekend admission, and seasonal influenza. Medical care. 2010 Mar; 48(3):224-32.(Link to article – subscription may be required.)
  29. NHS services, seven days a week forum. Summary of initial findings Dec 2013 NHS England
  30. Freemantle N, Ray D, McNulty D, Rosser D, Bennett S, Keogh BE, Pagano D. Increased mortality associated with weekend hospital admission: a case for expanded seven day services? BMJ (Clinical research ed.). 2015; 351:h4596.(Epub)
  31. Meacock R, Anselmi L, Kristensen SR, Doran T, Sutton M. Higher mortality rates amongst emergency patients admitted to hospital at weekends reflect a lower probability of admission. J Health Serv Res Policy May 6, 2016 doi:10.1177/1355819616649630
  32. Aldridge C, Bion J, Boyal A, Chen Y, Clancy M, Evans t, Girling A, Lord J, Mannion R, Rees P, Roseveare C, Rudge G, Sun J, Tarrant C, Temple M, Watson S, Lilford R. Weekend specialist intensity and admission mortality in acute hospital trusts in England: a cross-sectional study. Lancet 10 May 2016 DOI:
  33. Tooke J. Aspiring to excellence. Findings and final recommendations of the independent inquiry into modernising medical careers. Jan 2008
  34. D. Shape of Training. Securing the future of excellent patient care. GMC 2013