ACSM Fellow Publishes Bad Science on CrossFit and Paleo (Updated)

Jason and Rory's Pancake breakfast is "paleo" by this study's standards.

Jason and Rory’s Pancake breakfast is “paleo” by this study’s standards.

Devor and Smith Get the Band Back Together

If you want to study the effects of the Paleo diet, the Paleo intervention group should follow the Paleo diet. This shouldn’t be necessary to state. But it is.

Remember Drs. Steven Devor and Michael Smith? After failing to substantiate their first CrossFit study and getting hit with multiple lawsuits for fraudulent data, they’ve published another study on the athletes at CrossFit 614.

Their first study on CrossFit 614 alleged that 9 subjects dropped out “citing overuse or injury.” But the subjects, the gym owner, and the study’s own data coordinator all say the 9 subjects weren’t injured. And the researchers had no way of contacting the absent subjects to find out why they didn’t show up.

Devor and Smith’s second study is “Unrestricted Paleolithic Diet is Associated with Unfavorable Changes to Blood Lipids in Healthy Subjects.” It may be even worse science than the first.

Under “Dietary Intervention,” the study claims,

A Paleolithic diet, as first described by Eaton and Konner, was implemented for all study participants (11). Subjects were advised to increase their consumption of lean meat, fish, eggs, nuts, fruit, and vegetables and were instructed to strictly avoid all grains, dairy products, and legumes. All modern, processed foods including any form of processed sugar, soft drinks, and coffees were also excluded from the diets of the subjects.

Every line above is false. Let’s start with Line 1:

False Claim #1 “A Paleolithic diet, as first described by Eaton and Konner, was implemented for all study participants”

CrossFit 614’s challenge did not require full compliance with the Paleo diet. I asked CrossFit 614 owner Mitch Potterf if the subjects followed a strict Paleo diet for 10 weeks. He replied,

No. We gave points for several different behaviors ( men will die for points) and took away points for negatives. Ex)
1 Point-Paleo meal/snack (Pick it or kill it, no dairy, grain or processed food)
1 Point Zone Meal/snack ( We used a few CFJ articles to explain as well as books)
3 Point Paleo/Zone meal/snack
Negative 1 point for each Non Paleo/Zone meal
Negative 1 point for more than 2 Alcoholic drinks/week
Negative 2-5 pts don’t remember exactly for tobacco use

So the subjects could have followed the Zone diet only, and not Paleo. Or they could have done neither and simply took the hit in points.

False Line #2
“Subjects were advised to increase their consumption of lean meat, fish, eggs, nuts, fruit, and vegetables and were instructed to strictly avoid all grains, dairy products, and legumes.”

As seen above, CrossFit 614 did not instruct its athletes to “strictly avoid” non-Paleo food. In fact, the challenge encouraged consumption of grains, dairy, sugar, and other foods in cheat meals. Potterf states, “We also gave bonuses for the most epic cheat meal (AMRAP90min) caught on camera (winner got bonus points losers gambled and got negative points).”

One of CrossFit 614's epic cheat meals during the same period that Devor et al. claim they stayed strictly Paleo.

One of CrossFit 614’s epic cheat meals during the same period that Devor et al. claim they stayed strictly Paleo.

False Line #3
“All modern, processed foods including any form of processed sugar, soft drinks, and coffees were also excluded from the diets of the subjects.”

Even the athletes who did try to stick to mostly Paleo foods did not exclude coffee from their diets. It is strange that the researchers would even assume that Paleo dieters don’t drink coffee, the choice beverage of the Paleosphere.

Potterf’s own noble policy is that he “would never allow coffee to be removed from the diet of anyone at my gym.” As for the other fun foods, they were also allowed in the challenge, as long as the athlete admitted it and lost points.

In summary, Potterf doesn’t think a single subject followed the dietary intervention that Devor alleges:

I would say we probably didn’t have anyone go strict for more than a few weeks before they cheated. It really wasn’t a Paleo Challenge. The truth is nobody really knows what people actually did because they didn’t write down what they ate, just whether it could score them points or not.

Another CrossFit 614 member stays strict Paleo, according to Devor and Smith.

Another CrossFit 614 member stays strict Paleo, according to Devor and Smith.

How Did Devor and Smith Get it Wrong?

At this point it’s not clear why the researchers completely misrepresented this dietary intervention. I asked Potterf if they observed any of the subjects eating, ever. He responded, “No, not unless they were hiding in the bushes with a camera.”

If the researchers didn’t watch any of the subjects eating, perhaps they could’ve compiled diet logs during the intervention, right?

Nope. The researchers didn’t make that effort, either. They collected only 8 of the 44 subjects diet logs. But it gets worse. Those diet logs didn’t come from the actual intervention period.

A few months after the whole challenge Mike Smith asked me to hand out some diet logs and get people to record what they ate during the challenge. A few returned their sheets but they couldn’t really remember exactly what they ate so they ( Smith) just told them to write down a sample of an average day and they would use that for the whole thing.

So, will the researchers ever explain why they made these false claims? If Devor’s history is any sign, it may take a lawsuit.

The Results

So what did Devor conclude about the Paleo diet based on subjects on who did not follow the Paleo diet?

In conclusion, our study shows that the Paleo diet was significantly deleterious to blood lipid profiles in healthy subjects concurrently participating in a CrossFit- based, high-intensity circuit training program. Specifically, subjects with optimal initial blood lipid values demonstrated the greatest increase in LDL, TC/HDL, TC, and n-HDL values, along with the greatest decline in HDL values, following the 10- week Paleo diet intervention.

We don’t yet have any evidence, besides precedent, to doubt the blood lipid levels Devor and Smith collected. That said, this statement is also false. It asserts that the Paleo diet worsened the blood lipid levels of the subjects in that study. This could only be true if the subjects all followed the Paleo diet as described in this study. They didn’t.

Therefore, we can’t say what caused the admittedly surprising deterioration in blood lipid levels. Did they make too many attempts at winning the 90-minute AMRAP cheat meal challenge? One of the subjects admitted to eating 1-1.5 pounds of bacon a day, because it was Paleo. Perhaps this was a common behavior among the subjects, and it affected their blood lipid levels.

We can guess. But we really have no idea what was responsible for those changes. And neither do the researchers, though you’d never know that from reading the study. The accurate title for this study would have been “Varied Nutritional Approaches are Associated with Unfavorable Changes to Blood Lipids in Healthy Subjects.” But who would publish that?

Section 2 of the ACSM’s Code of Ethics states, “Members should maintain high professional and scientific standards and should not voluntarily collaborate professionally with anyone who violates this principle.”

I’m not aware of any scientific standard that tolerates repeatedly publishing inaccurate and unsubstantiated claims. If this behavior is not enough for the ACSM to revoke Steven Devor’s fellowship, what does it take? Where does the ACSM draw the line on scientific misconduct?

"90 minutes of gluttony (I felt sick posting this)... 4 glasses of cabernet, bread (lost count), melted pecorino cheese, apple and honey app, spaghetti carbonara, pizza, goat chese ravioli, dessert #1 tiramisu, dessert #2 cherry cobbler a la mode and because there were still a few minutes on the clock...some terrible Italian liquor chased with espresso....gross"

“90 minutes of gluttony (I felt sick posting this)… 4 glasses of cabernet, bread (lost count), melted pecorino cheese, apple and honey app, spaghetti carbonara, pizza, goat chese ravioli, dessert #1 tiramisu, dessert #2 cherry cobbler a la mode and because there were still a few minutes on the clock…some terrible Italian liquor chased with espresso….gross”

10 comments

  1. Landon

    As far as coffee is concerned – it should be avoided. The caffeine stimulates the adrenal glands which causes cortisol which causes fat storage. That is a fact. These young dudes doing CrossFit who drink coffee are pretty much immune to what I’ve said because being young, they have strong metabolisms. It’s the metabolism that is the key. Your average American, not working out, eating crappy food, not sleeping enough, too stressed out – cannot be drinking coffee; trust me.

    • Turkish

      Cortisol doesn’t cause fat storage without a catalyst, like insulin. Most CrossFitters drinking coffee, are drinking it alone and if it isn’t black it’s with cream, butter or MCT oil.
      Cortisol helps store fat in most Americans due to a high carbohydrate diet.

      • bane73

        Well I don’t know about you, Turkish, but I feel so much better about Landon’s claim… you know, now that he’s “assured” us that he’s correct.

        LMAO.

        Landon, I “assure” you, I found your argument very persuasive.

  2. Christopher Haddock

    From a methodological point of view the study was unimpressive. Aside from the issues you have already noted (e.g., lack of standardization, lack of monitoring, etc), the research design was poor and the statistical analyses were unimpressive. One of the worse research designs you can select in terms of causal inference is a one group pretest-posttest design as in this study. It is so fraught with threats to internal validity it is almost not worth doing. Probably the reason they had to publish it in the journal they did and not in a tier 1 or 2 outlet. Also, the statistical analyses almost guarantee they will find some significant results. If you do lots of analyses on pre-post changes using paired t-tests, including multiple tests on subgroups, its not surprising that some of the p-values will be < .05. No statistician will become giddy over a significant within group t-test, particularly when lots are done.

    • Landon

      You, apparently, are not that familiar with the effects of cortisol. It is a factor in belly fat along with other inputs (and it does promote insulin). Young dudes doing CrossFit with their inherent strong cells and mitochondria are not going to be affected by caffeine in this manner – or by poor eating habits, per se. When one is older many factors involving nutrition CAN have a negative effect on someone’s body composition and health.

      • bane73

        But, Landon, that all is well and true I’m sure… but unless and until you “assure” us, we are left uncertain.

  3. Pingback: The Roundup

  4. Pingback: FC Team Week Updates, AMRAP90 Epic Cheat Meal | Fit Club

Comments are closed.

%d bloggers like this: