The Washington Post posted an article today by the well-known and highly regarded horse racing columnist Andrew Beyer titled “Thoroughbred Racing Under Fire After Investigative Reports, Cancellation of ‘Luck.’” Mr. Beyer made an astute observation about the New York Times expose on horse racing that calls into question the methodology used by the authors and hence their conclusions:
“The Times focused on racing in New Mexico, but readers undoubtedly assumed that the horrendous breakdowns and injuries to jockeys in that state were mirrored in New York, home of the country’s top thoroughbred racing.
However, almost all of the New Mexico horror stories cited by the Times occurred in quarter-horse racing — a different sport, with a different breed, a different style of training and a different ethic… According to the Times’ own statistics, the seven U.S. tracks with the highest percentage of breakdowns or signs of injury were all ones that offer quarter-horse racing — five of them in New Mexico, where supervision was notoriously lax. Yet the Times never drew a distinction between the two sports and did not even mention the phrase ‘quarter horse’ until the 48th paragraph of its report. Subtract the quarter-horse component from the study and the Times might not have a carnage-laden front page story.”
Mr. Beyer has perceptively brought up the subject of “external validity.” When a survey researcher inquires about “external validity,” he or she is asking whether results obtained from a sample can be projected to make predictions about the entire population from which the sample is drawn.
The Times study does not have external validity in extending the results from its study to Thoroughbred racing (or to harness racing) for reasons pointed out by Mr. Beyer; i.e., the sample included “the seven U.S. tracks with the highest percentage of breakdowns or signs of injury were all ones that offer quarter-horse racing — five of them in New Mexico…”
I am going to assume that the authors did not know the proper sampling procedure and erred, rather than did know and glossed over it to sensationalize the story and get it on the front page.
The Times authors should have acknowledged that their study is exploratory in nature rather than leaving the impression that they followed the scientific research procedures that would have enabled them to make the hard-and-fast assertions in the article.
_____________________
The following was released today by the office of U. S. Senator Tom Udall of New Mexico:
Udall: Times Investigation Paints Disturbing Picture of Horseracing Industry
WASHINGTON – Following an in-depth report by The New York Times on the state of horseracing in the United States,U.S. Senator Tom Udall (D-NM) issued the following statement:
“The findings uncovered in The New York Times investigation about horseracing in the United States, and New Mexico in particular, paint a very disturbing picture of the industry.
“The sport of horseracing which, at its best, showcases the majestic beauty of this animal and the athleticism of jockeys, has reached an alarming level of corruption and exploitation. The consequence of inconsistent state-level regulation is an epidemic of animal doping that has led to countless euthanizations of helpless horses and the injury and death of their riders.
“The Times exposé has shined a glaring light on the need for national standards in a sport that reaps gambling profits, but has lacked proper oversight for decades.
“I urge our leaders in Congress to advance the bipartisan legislation Congressman Ed Whitfield and I have introduced in both chambers to renew the sport of horseracing and set minimum, nationwide standards for medication and doping. The Interstate Horseracing Improvement Act would kick cheaters out of the sport. The horseracing industry has promised voluntary reforms for decades, but as we’ve painfully observed, our legislation is the only viable way to address doping problems plaguing the sport.
“Now is the time to end the unscrupulous practices of those trainers and track veterinarians in horseracing who abuse these magnificent animals and endanger jockeys for gambling profits.”
The Times study would be rejected by a peer-review academic journal for the reasons you and Beyer point out. Trouble is readers will take the results as gospel.
Your point about the lack of external validity probably won’t be understood by most, but it is correct and pretty well skewers the Times study. QH racing in NM can’t be used to make inferences about TBs. Beyer makes the same point in so many words.
My intent in writing about the Times article is to judge its content from a factual and scientific basis. As a piece of research it does not pass muster in terms of the kinds of generalizations the authors are trying to make. In addition to choosing the wrong sample to make projections to Thoroughbred racing, the sample size is way too small and is anecdotal.
I believe the authors to be fine and successful people and I am not going after them as individuals, but rather, am only pointing out what I see as weaknesses in their approach. I spent my working life doing and evaluating research, both in academics and as a consultant. I have had my own work evaluated by many other professors in double-blind reviews and I rarely took their comments personally. That is the same spirit with which I criticize the work by the Times’ authors. Further, I have no vested interest, as my livelihood does not come from horse racing.
I also believe that something useful can come of their work, even with the noted research flaw of improper sampling for making generalizations about Thoroughbreds.
The Times is perpetuating the lie. Tuesday’s article never mentions quarter horses, only talks about thoroughbred racing. This implies that the results from the New Mexico quarter horse races are from thoroughbreds, which is not true at all. This is outright deception. Drape should be ashamed of his part in this. The NYT does have an agenda.
Forgot to say that after today’s Times story it is evident that your assumption about the authors’ motives is wrong because they have been told about their errors:
“I am going to assume that the authors did not know the proper sampling procedure and erred, rather than did know and glossed over it to sensationalize the story and get it on the front page.”
This begs the question why the NYT did not hire some experts in designing scientific studies to advise them. I think we know the answer. Sampling expert to Drape and others…”your QH sample won’t support your conclusions about TBs.”
Reply: “OK, thanks, your services are no longer needed. Let’s go with the article, a little deceit won’t hurt anyone. It is just a lie for the greater good.”