Vaccine Science Review #1

Official site:  http://www.freedom2think.com/vaccinescience

by Helen Tucker

Article reviewed:
Barlow, W. et. al.  2001.  "The Risk of Seizures after Receipt of Whole-Cell Pertussis or Measles, Mumps, and Rubella Vaccine," New England Journal of Medicine 345: 656-661

[A free copy can be obtained from http://www.nejm.org/ .]

Review abstract:

Barlow et. al. studied data from four HMO's on the west coast to assess the risk of seizures and neurobehavioral disorders after receiving the DTP and MMR vaccines.  They reviewed 624 medical records out of an undisclosed number of children using undisclosed methods and misleading control groups to calculate meaningless risk coefficients.  They then concluded there were only small, "transient" seizure risks associated with DTP and MMR which "do not appear to be associated with any long-term, adverse consequences."

A.  What did the study do?

The authors said they studied nearly 680,000 children aged 0 - 7 who were enrolled at these HMO's during the observation period, which was different at each HMO, but ranged between 1/91 to 9/93, with some follow-up data extending to 1998.

The authors concluded that there was a slightly increased risk of febrile seizures on the day of vaccination with DTP, which is estimated to mean 6 to 9 additional febrile seizures for 100,000 children.  They also found a slightly increased risk of febrile seizures in the second week after the MMR, which is estimated to mean 25 to 34 additional febrile seizures for 100,000 children.  They concluded there was no increased risk of  nonfebrile seizures, subsequent seizures after the first febrile seizure, and neurobehavioral disorders for children receiving the DTP and MMR.

B.  What were the flaws of the study?

1.  Meaningless sample:  The authors failed to provide sufficient evidence that their sample was valid and representative of the population they were studying. 

Although the study started out with almost 680,000, the authors excluded anywhere from around 42,000 to 135,000 children, without a clear explanation.  The sample size was narrowed by 42,000 at the beginning to about 638,000 "person-years of observation" without any definition of this term.  It was also narrowed by 13.7%--of which number, they never said--because of disenrollment from the HMO's. 

From whatever figure they ended up with, they found 2281 possible seizures from highly variable sources depending on the HMO, anywhere from hospitalization records to neurology referrals.  They never explained how many seizures came from each source.    From the 2281, they sampled 1094 for risk analysis.  Because sampling proportions varied from HMO to HMO, the authors weighted the data from some random samples to make them comparable to the rest.  They did not say how many seizures came from each HMO and how many cases were weighted.  The authors admitted the sample was somewhat biased toward severe, hospitalized cases, but failed to present information necessary to objectively assess the extent of the bias. 

From the 1094, they whittled the final sample down to 624 children on which they presented results of their analysis.  They excluded children because seizures happened outside of the study period or before the inception of a vaccination database, but never defined those dates.  They excluded children because of specific infections or injuries, but never defined them.   They presented a category of neonatal seizures, but never discussed them in the analysis, so it was even unclear who exactly were excluded.  Without the objective criteria by which subjects were excluded, there is insufficient evidence to determine that the sample was not further biased.  It was the authors' responsibility to justify their sample as valid and representative, and they failed.

2.  Meaningless and Misleading Control Group:  The authors compared children who were recently vaccinated to children who were not recently vaccinated to conclude that the risk of vaccines over the "absence of vaccination" was small.

To assess the risk of seizures among children who had received the DTP and MMR, the authors needed to compare children "exposed" to DTP and MMR to children who were "unexposed."  The exposed group was defined as children who had had either the DTP, the MMR, or both vaccines concomittantly in 30 days preceding the first seizure episode.  The unexposed group was defined as children who had not had either vaccine in the last 30 days.   The authors implied, but never explicitly stated, that the "unexposed" group was also unexposed to any other vaccine in the last 30 days before seizure.  The control situation was referred to as "absence of vaccination."

The authors did not provide evidence that a 30 day difference between the exposure to vaccines in the two groups is sufficient time to meaningfully distinguish between presence of vaccination and "absence of vaccination," exposure and "nonexposure."  The authors needed to carefully qualify that their risk calculations were only for children who were recently vaccinated with DTP/MMR compared to children who were not recently vaccinated.  They needed to qualify that they were in all likelihood comparing vaccinated children to vaccinated children.  They did neither.

The authors also wanted to assess the seizure risks of DTP and MMR separately.   The fact that some children received both DTP and MMR in the last 30 days before their seizures confounds the risks attributed to either vaccine individually.  The extent to which the variables were confounded is unknown because the authors did not say how many received both vaccines. 

Finally, the authors admitted that vaccination status was not always accurate.  They did not explain how many were inaccurately identified as vaccinated or unvaccinated, and how they resolved disagreements.

3.  Meaningless and Unreliable Risk Calculations:  The authors presented seemingly arbitrary numbers with apparently large errors and used invalid background rates as the foundation of all their conclusions.

The authors concluded that certain risks were small and other risks were non-existent.  The only evidence provided to support these conclusions were calculations of "relative risk."  The authors said they used the "stratified Cox proportional-hazards analysis" and "standard methods" to calculate these numbers, but did not define what these methods were, provide equations, provide literature references, or present any other information that would validate these calculations.  Furthermore, these relative risks were calculated after "adjustments" for selected variables.  Although they defined some of these variables, others were unclear and how the calculations were "adjusted" was not explained. 

The only data presented was each "relative risk" number and the 95% confidence interval, such as "5.70 (95% CI, 1.98 to 16.42)."  They did not give any meaning to these numbers, such as standard deviation, relative error, statistical significance, or any other contextual information.  As presented, the relative risk numbers are completely meaningless.

The confidence intervals they did provide showed that these calculations were highly unreliable.  For example, the risk of "5.70 (95% CI, 1.98 to 16.42)" translates to mean that 95 times out of a 100, their calculation could range anywhere from 2 to 16. 

In addition, these relative risks were compared to two "background rates" of febrile seizures to determine how many additional seizures per 100,000 children can be attributed to DTP or MMR.  One of the background rates came from the same data as the risk calculations, so it is not surprising that there was only minimal difference between the two.  The other background rate came from a 1969 published study of one of the participating HMO's, with data more than 20 years old.  The absence of more current and broader background rates makes the number of additional seizures attributed to DTP/MMR meaningless.

4.  Meaningless Follow Up.  The authors followed whichever children they wanted, for only the diagnoses they wanted, then calculated meaningless numbers after "adjusting" the data, to yield the conclusion that the vaccines were not associated with "long-term, adverse consequences."

Out of the 74 children who had febrile seizures within 30 days of vaccination, the authors followed only 40 to study any increased risk for subsequent seizures.  For the DTP, they followed only children who had febrile seizures in the first 7 days.  For the MMR, they followed only children who had febrile seizures within 7 - 21 days.  They did not explain why they followed only these children and not all of them.  They did not follow any children with nonfebrile seizures after vaccination because the authors believed there was no association between the two events.  The control group consisted of 521 children who had febrile seizures in the "absence of vaccination," that is, no DTP/MMR in the preceding 30 days.   The authors did not explain how long these children were followed, though they suggested that some of the children were followed for at least 2 years.  Nonetheless, they concluded the 40 exposed children were no more likely to have subsequent seizures than the 521 "nonexposed" children. 

Out of the 561 children followed for subsequent seizures, 273 were followed for neurobehavioral diagnoses.  Again, the authors do not explain how the 273 were followed, for how long they were followed, and why they excluded the rest.  The diagnoses they flagged included:

"ADD, learning disorders, mental retardation, speech disturbances, emotional disturbance, personality disorder, repetitive-movement disorder, obsessive-compulsive disorder, infantile autism, childhood type schizophrenia, and psychoses of early childhood."

Notably, with the exception of infantile autism, they did not include any of the Pervasive Developmental Disorders considered to be in the autism spectrum. 

The authors concluded that the risk of these conditions did not differ between exposed and unexposed children, after an "adjustment" for age at the time of the seizure.  This implies that there was a difference between exposed and unexposed without the adjustment.The only evidence cited to support this conclusion was again, one "relative risk" number.  Again, the enigmatic sampling, poorly defined control group, narrow selection of long-term consequences, and meaningless risk numbers provide insufficient justification to give these conclusions any meaning or validity.

C.  What can be concluded from this study?

The only results that can be credibly presented are the tabulation of incidences. 

1.  Out of an unknown number of children, at least 42 who received the DTP vaccination had febrile seizures within 30 days, and at least 10 had nonfebrile seizures.

2.  Out of an unknown number of children, at least 32 who received the MMR vaccination had febrile seizures within 30 days, and at least 3 had nonfebrile seizures.

3.  Out of an unknown number of children, at least 413 who had febrile seizures did not receive either the DTP or MMR in the preceding 30 days.  At least 124 children who had nonfebrile seizures did not receive either the DTP or MMR in the preceding 30 days.

Because this paper fails to meet minimum scientific standards for the presentation of objective evidence, no other conclusions are defensible. 

------------------------------------

The author would like to thank the Vaccine Science discussion list for their input and feedback.

http://groups.yahoo.com/group/VaccineScience/

Permission is granted to forward or reprint this article on the condition that it is reproduced in its entirety without any changes (everything between and including the 2 asterisk lines).   If this article is reprinted on another website, the author would appreciate a note with the link to the site.  Her email address is list@freedom2think.com.

Review by Dr. Cranky

Barlow, W. et. al.  "The Risk of Seizures after Receipt of Whole-Cell Pertussis or Measles, Mumps, and Rubella Vaccine," N Eng J Med 345: 656-661 (2001).

If you wanted to find out if Windex made rats vomit, how would you go about it?  Well, you'd take a bunch of rats, and randomly divide them into 2 groups.  Feed Windex to one group, and water to the other.  Watch them for a few days and count how many rats vomited in each group.  Find what percentage of Windex-drinkers vomited vs. water-drinkers, and see if the difference is significant, right?  It ain't rocket science.

Barlow and his buddies apparently had time to waste.  They searched thousands and thousands of rats to find some that vomited.  Then they asked around to see which vomiters happened to have had some Windex for lunch.   Some of the eyewitnesses disagreed, but hey, nothing's perfect.  They figured most of the vomiters didn't have Windex for lunch and concluded that Windex is just peachy to feed to rats. Feel like investing in a Windex bar for rats yet?

The authors wanted to find out if the DTP and MMR vaccines caused seizures.  They found some kids with seizures, and decided that most of them didn't have the DTP or MMR in the last month, and therefore DTP and MMR aren't linked to seizures--much.

As a university professor, if someone turned in a report like this, I would slap down an F before I finished reading the first page.  I would ask some serious questions.  Do you have any other data besides sample size?  After reading four pages on practically nothing but sample size, I still don't know how many people you started out with.  Where are your equations?  How about showing your work, or any justification for your numerical results?  If you're pulling numbers out of thin air, why did you pick such crappy numbers?

I have rarely seen so much bullshit all in one pile, which usually means one thing:  obfuscation of the fact that for all that time and money spent, they don't have anything to tell us but bullshit.  Or worse.  Did they analyze thousands of records, only to find numbers that didn't speak favorably of vaccines?  Did they then eliminate and weight and massage the numbers to yield the results they were hoping for?  Is that why they don't give details of anything they did?   Did they use big words and circuitous writing to cover up for missing and tampered data?  Hoping that no one would want to spend the time and effort to disentangle their complete mess of a paper? 

Either way, if you're looking for insight into vaccination risks by reading this article, you're wasting your time.

[You can forward this review to anyone you want, provided nothing is changed.  Anyone got complaints, write me at drcranky@att.net.]