Which Have Been the "Best" Events This Season?

If you've been reading my blog for any length of time, you know that one of the ways I like to evaluate the effectiveness* of a CrossFit event is by looking at how well the results from that event correlate to results from a variety of other events. I laid out the theory behind this in my post from last year titled "Are certain events 'better' than others?", but I'll recap it here:

• In a competition setting, what we are trying to do is learn as much about each athlete's overall fitness level as possible.
• We only have a limited number of events to do this, particularly in the Open and at Regionals. Therefore, we need to maximize the information we get from each event.
• If athletes who score well on a particular event tend to score well across the board, then that event is probably a good indicator of overall fitness. Conversely, if the results from that event don't correlate at all with results in other events, then maybe that particular event did not really tell us much.

Overall, this year's regionals and Open were set up well for me to do this type of analysis. I did this same analysis after last year's regionals, but due to the cuts after regional event 5, I was left with only about 250-300 athletes of each gender who completed all the Open and Regional workouts, and this limited the analysis to only the very elite athletes. I also did this analysis after the Open this year, which gave me a huge sample of athletes, but with only 5 events, I didn't really have a "wide variety" of events to evaluate. 

This year, because we did not have any cuts at Regionals, I was left with 673 men and 512 women who have completed all 12 events this season. Although these are all still very solid athletes, I got a lot more of the borderline regional competitors in the mix than I did last year. Remember, a "good" event for the Games might not make a "good" event for a competition within your own box. So keep in mind with this analysis that we are evaluating these events based on how well they predict overall fitness for Regional competitors.

The methodology for performing this analysis this year was the following:

• For athletes who completed all events this year, compile their results (not just their ranking) for each of those events. The Regionals results I used are those that have been adjusted to account for the week in which week each athlete competed. See my previous post for more info on those adjustments.
• Rerank the entire field in each of those events.
• For each event, calculate the sum of each athlete's ranks in all other events.
• Calculate the Pearson correlation (referred to from here on out as simply "correlation") between the ranks in each event and the sum of ranks in all other events. Higher correlations indicate "better" events.

Below are the results for both men and women.

The pattern that emerges is one I've noticed pretty much across the board since I've been doing this type of analysis: events with more movements tend to be better tests of fitness. This makes sense intuitively, since they test more things, by definition. That doesn't mean we shouldn't have single-modality events in competition, it just means they probably should be used sparingly and only for movements that are deemed very important.

You may notice that Open Event 2, which had three movements, is bucking this trend by falling quite close to the bottom. The concerns with this event were pretty well documented during the Open: judging was very difficult, the weights were extremely light for top competitors, and the option of step-ups was utilized by a lot more athletes than was probably expected. So simply because you have 3 or 4 movements in an event doesn't make it a great one.

To get a visual interpretation of the concept I'm getting at here, below are scatter plots for some of the best and worst events from this year. On each graph, the x-axis represents each athlete's rank on that event and the y-axis represents each athlete's combined rank on all other events.

It should be fairly clear that for the first two graphs, there is a clear relationship between the x- and y-axis. Athletes who did well on these events generally did well across the board. On the third graph, the points are much more scattered, indicating that there were plenty of athletes who did well on Open Event 2 who didn't fare well across the board, and vice versa.

Although I consider the results using the entire Regional field to be the most useful, I also tested performed this analysis on three other subsets of the regional field:

1. Games athletes only
2. Top 292 men and top 258 women (same number as I had in my analysis last year)
3. Random sample of 20% of the entire regional field

What I was interested in was how volatile my results were. For instance, is Regional Event 1 really better than Regional Event 4 for women (76% vs. 74%)? Would that hold up if I changed the group of athletes a bit?

In general, the events near the very top and the very bottom stayed in that vicinity, with a couple of exceptions. Here are the main takeaways:

• For men, Regional Event 4 was in the top 3 across all the samples and Open Event 1 was in the top 5 across all the samples. For women, Regional Event 4 was in the top 4 across all the samples, Regional Event 1 was in the top 5 across all the samples, Open Event 1 was in the top 5 across all the samples and Open Event 4 was also in the top 6 across all the samples.
• Considering HQ is programming the same events for both men and women, I would conclude that Regional Event 4 ("The 100s") and Open Event 1 (AMRAP 17 of burpees and snatch) were the best events this season. In one of my 2013 Open recap posts, I noted that 13.4 was generally the best event of the Open. I still feel that it was a very good event for the entire Open field, but it wasn't quite as strong when we look at just these stronger athletes. 
• Across both men and women, Open Event 2, Regional Event 5 and Regional Event 3 were each in the bottom 4 in all but one sample. I would conclude that these were generally the three weakest events this season. 
• I mentioned some issues with Open Event 2 above.
• For Regional Event 5, I think the issue is that we saw a lot of athletes near the bottom of the field do well on this simply because they could deadlift a house. If you could handle the deadlifts easily, you could generally do well even if you weren't particularly great at box jumps or had sub-par aerobic capacity.
• For Regional Event 3, I think the issue was that the burpees did not really factor into this much, making it basically just a muscle-up test. As far as single-modalities go, this wasn't too bad of an event. But personally, I think this event and the overhead squat ladder (a true single-modality) should have been worth only 50% of the other events.  
• Two of the events featuring box jumps turned out to be relatively modest tests of fitness. Throw in all the complaints about achilles problems we've seen popping up recently, and I think HQ may want to look into adjusting how they program box jumps. I think box jumps are a good test of fitness in general, but I'd personally love to see us go to box jump-overs (onto and over the box with an option to jump straight over) in the future.
• With the exception of Open Event 2 and Regional Event 5, I think the rest of the events were generally solid. As I mentioned above, I might consider adjusting the point value for a couple of the other ones.

I also did one final analysis, primarily out of curiosity. Using the entire Regional field, I looked at the correlation between each pair of events. Some of the interesting findings:

• In general, the most highly correlated pair was Regional Event 4/Regional Event 1 (71% for women and 67% for men). This is somewhat surprising given how the time domains were completely different, but both involved pull-ups and a light thruster-type movement (bar thrusters or wall-balls).
• The two muscle-up workouts (Regional Event 3 and Open Event 3) were highly correlated (68% for women and 55% for men).
• Regional Event 5 and Regional Event 7 were highly correlated (56% for women and 68% for men). Both were extremely heavy.
• The least-correlated pair was Regional Event 3/Regional Event 5 (17% for women and 19% for men). Shouldn't be a surprise considering one was bodyweight only and one involved extremely heavy deadlifts. The pair of Regional Event 2/Regional Event 3 were also not very correlated (38% for women and 22% for men). Remember those occurred within 2 minutes of each other.

That's about it for today. This is always one of my favorite analyses to work on, but with the Games fast approaching, I suppose it's about time to start tackling the tough questions and making some predictions. Will Froning three-peat? (Probably) Who will emerge on the women's side with Annie out? (It's wide-open) Will the first event of the Games take more or less than 4 hours? (God, I hope so) What bizarre contraption will Rogue unveil this year? (Potentially a flying bicycle, similar to the one in E.T.) Who will wear the shortest shorts this season? (Stacie Tovar still the champ until proven otherwise)

Anyway, until next time, good luck with your training!

*I am referring to the effectiveness of this event as it relates to competition. In other words, is this event a good test of fitness. This does not necessarily mean the event is good or bad for training purposes. For instance, I feel that 13.2 was not a good workout for testing (due to a lot of factors, like how difficult it was to judge and how light it was for the top competitors), but in training, I think it would be a good workout for building aerobic capacity (and it definitely left me hurting).

A Fairer Regional Comparison: 2013 Edition

Today's post will mark an anniversary of sorts for my blog. Roughly a year ago, I put up my first post, in which I produced adjusted worldwide Regional rankings, accounting for the advantage gained by athletes in the later weeks. The post went up with little fanfare, and I imagine many people who started coming to this site via Rudy Nielsen's Outlaw Way blog may not have even read this original post. In hindsight, it was probably overly technical, but give me a break: it was my first post.

Well, today I'll be doing essentially the same thing for this year's regionals. But first, let me clarify what I believe these rankings really mean. They are not my predictions for the Games, and they are not a ranking of the best CrossFitters in the world. Rather, what they are is an attempt to understand who would win if the exact same Regional events played themselves out at the Games. If all the Games qualifiers competed together, at the same time, using the Games scoring system, who do we believe would win? Certainly this gives us a good starting point to discuss who may win the Games (hint: Rich Froning has a shot!), but it's not a prediction for the Games.

Many of you have probably seen a version of these rankings at http://crossfitregionalshowdown.com/leaderboard/men and http://crossfitregionalshowdown.com/leaderboard/women. These are certainly informative, and in fact I grabbed all the original results from this site, so I really appreciate the work they've done. But my feeling has always been that the athletes in the later weeks have an advantage that isn't captured in these rankings. It stands to reason that having additional weeks to prepare, watch other competitions and game plan the events has to give athletes some advantage.

To get an idea of the advantage across the entire field, I ranked all athletes that finished all 7 Regional events based on their worldwide Open rank. Then I ranked them by their worldwide Regional rank (from the sites listed above). For each region, I looked at the average change from Open rank to Regional rank, then plotted them along with the week in which each region competed. This is shown in the chart below, with lower numbers representing improvements from the Open to Regionals.

You can see that while there is variation, the latter weeks clearly tended to have better scores from athletes, relative to their Open performance. This isn't a perfect metric: for one, some elite athletes don't take the Open as seriously as others, but also, it gets skewed at the each end because the top athletes in the Open can only get worse and the worst athletes in the Open can only get better. Still, this tells us that something is probably going on. For women, the same effect is there, but it's not quite as pronounced.

To attempt to account for this, I performed a series of two-variable linear regressions. For each event, I used the event result (the actual result*, not the ranking) as the dependent variable, and for one of the independent variables, I used the week of the athlete's region. For the other independent variable, I used either the athlete's 2012 Regional ranking (if he/she competed individually in 2012) or the athlete's 2013 Open ranking (if he/she did not compete individually at Regionals in 2012). Then I looked at the coefficient for the week of competition - this would give an indication of the impact of the week of competition, after controlling for the athlete's ability.

I then divided the coefficient by the average result on that event for all athletes to get the percentage impact of the week of competition. Depending on some of the summary statistics and using a bit of judgement, I arrived at a final adjustment factor for each event (some were 0 if the results didn't appear significant enough). If the adjustment factor for event 1 was -1.0%, then for every week beyond the midpoint (week = 2.5), I adjusted the athlete's score up by 1.0%. If the athlete competed in week 4 and his score was 6:00, then his adjusted score becomes (6:00 / (1 - 1.5%)) = 6:05. If the athlete competed in week 1, his adjusted score becomes (6:00 / (1 + 1.5%)) = 5:55.

Below are the adjustment factors I used for each event for men and women:

• Event 1: -0.6% (men), -0.5% (women)
• Event 2: +0.7% (men), +0.8% (women)
• Event 3: -0.7% (men), none (women)
• Event 4: -0.2% (men), none (women)
• Event 5: none (men), none (women)
• Event 6: -1.5% (men), none (women)
• Event 7: -1.8% (men), -2.2% (women)

As you can see, these are not particularly aggressive adjustments. I think with the first wave of athletes having several weeks to prepare, the advantage is not dramatic each week beyond that. And for women, certain movements like muscle-ups and handstand push-ups are simply so troubling for many athletes that no amount of game-planning in a few weeks can make a significant difference.

For those who care, a couple notes on differences from my modeling last year, as well as some limitations:

• This year, I used all Regional athletes. Last year I only used those who completed all made the cut to the final event, but since there were no cuts this year, I used everyone.
• This year, I performed the regression across the individual athletes. Last year, I summarized up to the region level first before performing the regression. I think this year's method is preferable; last year I summed up first because I didn't have access to everyone's Open results, so I just counted up the number of athletes in each region in the top 180 worldwide as a proxy for region strength.
• This year, I included Asia, Africa and Latin America in running the regressions. Last year, I excluded them.
• I made no adjustments for the weather conditions at the outdoor regions. I realize that the elements may have made things more difficult, but with only a couple regions competing outdoors, it is difficult to assess just how much impact this had.
• I used no tiebreakers. If two athletes tied, they just stayed tied. Sue me.

OK, let's cut the chit-chat and get to the results. The tables below show my adjusted worldwide Regional rankings, along with the rankings if I had not made any adjustments for the week of competition and the rankings if I had not made any adjustments and used Regional scoring.

My adjustments were generally smaller on the women's side, so there wasn't a ton of shifting on that side of the leaderboard. For the men, we definitely saw some big jumps. Among the biggest winners with my adjustments were:

• Josh Bridges (7th to 4th)
• Lacee Kovacs (28th to 18th)
• Travis Mayer (32nd to 23rd)
• ZA Anderson (34th to 24th)
• Mikko Salo (30th to 22nd)
• Daniel Tyminski (18th to 14th)
• Marcus Filly (26th to 20th)
• Valerie Voboril (20th to 17th)
• Kristan Clever (31st to 28th)
• Katrin Tanja Davidsdottir (35th to 32nd)
• Talayna Fortunato (14th to 12th)

That's all for now, folks. Next week, I plan to look into all 12 events we've seen so far this season and see which ones were "better" than others. Until then, good luck with your training.


*For events 3, 4, 6 and 7, I added additional time for each rep still left at the time cap, since the 1 second per rep they have listed is not realistic. For events 3 and 7, I added 15 seconds/rep. For event 6, I added 10 seconds/rep. For event 4, I added 5 seconds/rep.

Quick Hits: Regional Reaction and Upcoming Schedule

It's been quite a ride the last four weeks of regionals, and to be sure, there is plenty of analysis to be done before we get to the Games in a few weeks. But for now, I just want to get some general thoughts together about the Regionals and my experience watching the Central East. I'll also look briefly at how my projections turned out, as well as what I plan to post over the coming weeks.

Let's get started:

• Watching the Central East regionals this past weekend solidified in my mind that CrossFit as a sport has staying power. The stands were legitimately packed all weekend, and some of the roars from the crowd were unlike anything I've heard at a CrossFit event before. In particular, one moment that stands out was the end of men's event 6 with Rich Froning chasing down Dan Bailey on the lunges. When Dan put the bar down halfway through, the crowd just sensed that Rich had a shot, took their energy to another level, and of course, Rich delivered, gutting out the entire 90 feet to take the event. Few other sports let the spectators really see that type of will and determination to endure pain, and on a regular and repeated basis. It's one of the reasons I've always enjoyed watching track and field, but I think the variety of CrossFit pushes the athletes to places they have never been to before. I think any fan of sport can appreciate seeing that.
• There's no question that the Central East men's region was hands-down the most competitive region. What is up for debate is whether HQ should do anything to remedy the situation. I think inviting athletes solely based on a worldwide regional ranking is not a smart idea - not unless the Regionals are all held on the same weekend and under identical conditions. That means no more outdoor events, and it probably means we all get to watch less of the action. But perhaps some sort of last-chance qualifier might be worthwhile for those in the top 10 of their region? Seeing athletes like Nick Fory and Gerald Sasser miss out on the Games despite putting up scores that would win several other regions is frustrating (see http://crossfitregionalshowdown.com/leaderboard/men).
• After making some adjustments based on the actual results that came in for the overhead squat workout, I recalculated the average relative weight and the load-based emphasis on lifting (LBEL) for this year's regionals. For both men and women, the average relative weight was higher than both 2011 and 2012, but the LBEL was lower. The way I interpret that is that the minimum strength required to be competitive at regionals was very high this year (high relative weight), but for those that could handle the required weights, the competition generally favored the smaller athletes (low LBEL).
• I'd still like to see running more involved in the regional competition. I know there are logistical issues, but it seems silly that in selecting the fittest athletes in the world, no one is required to run more than 800 feet (and even that 800 feet of running, in event 7, is little more than a light jog to recover). How about at least some shuttle sprints? Or a longer run on Friday when fewer people are able to attend anyway?
• On the men's side, it's going to be almost impossible to pick against Rich Froning for the Games. The guy is an absolute machine with virtually no weaknesses. Unless every event is under 4:00 or over an hour, I don't see a lot of scenarios where he doesn't win.
• For women, things are wide open. If Annie comes back, she's got a great shot (after the inevitable backlash against HQ for inviting her). Either way, Sam Briggs and Camille Leblanc-Bazinet look awfully solid all-around, and some of the stronger athletes like Elizabeth Akinwale and Linsday Valenzuela have become more well-rounded in the past year. There are others in the mix as well (Talayna Fortunato, Amanda Goodman, Kara Webb, Rebecca Voigt, etc.).

As for my predictions last week, things turned out fairly well. I totally whiffed on two athletes who made the Games (Lindy Wall and Jordan Troyan). For Wall, at least I have an excuse: last year, she was Lindy Barber, so I did not account for the fact that she was 6th at regionals a year ago. Had I known that, she would have been given a 16% chance instead of the 6% I originally gave her (I didn't even list her among top contenders on the site). With Troyan, although he was 10th at Regionals last year, he didn't have a great Open performance this year (20th in his region), so I just didn't see that one coming.

If I give myself a pass on that, then I ended up with a mean square error of 0.044 across the final two weeks (I predicted a total of 8 regionals). This was significantly better than any of the "default" estimates I mentioned last week. The best of those estimates would have been giving my top 3 athletes in each region a 100% shot, which would have produced a MSE of 0.052. There's always room to improve this modeling, but I think it was a decent start in my first year attempting to predict the regionals.

OK, finally, here's a quick listing of topics I plan to touch on between now and the start of the Games:

• A look at all the events thus far to see which are most correlated with success across a wide variety of events (i.e. which events are "better"). I did a very similar analysis last year and did a shortened version of it after this year's open.
• Adjusted worldwide regional rankings to account for the advantage gained by competing in a later region (and possibly other factors). This will be very similar to my inaugural post on this site approximately a year ago.
• Games predictions. I'll definitely do some sort of stochastic prediction to give each athlete odds of winning the Games (and potentially the odds of finishing second, third, etc.). I may separately do a best estimate of who will finish in the top 5 or 10.
• Perhaps a scientific wild-ass guess (SWAG) about some of the events we could see at the Games. This seemed to be pretty popular during the Open, and of course it requires little-to-no hard work on my part.
• Maybe a bit more analysis on the Open now that I've got a more complete dataset (thanks again to Michael Girdley at girdley.com for hooking me up with that data). This could wait until after the Games, depending on time.

If you've got other ideas, by all means let me know. Looking forward to the Games in just six short weeks. I've got my gold ticket, and I hope you all do, too!

Week 4 Predictions: Let the Big Dogs Eat

OK, so I may be a little biased living in Indianapolis, but I think I'm justified in saying that the Central East men's competition is really the crown jewel of the Regional season. I'll be headed down to Columbus, OH to watch Saturday and Sunday in person, and I have to say I'm pretty pumped. The numbers for this region are staggering:

• The winner of the last three CrossFit Games has come from the Central East (Graham Holmberg 2010, Rich Froning 2011-12). Both men are competing this weekend.
• The Central East produced five men in the top 10 of the 2012 CrossFit Games. All five are competing this weekend.
• Two other men competing this weekend qualified for the Games in 2011: Nick Urankar and Joseph Weigel. Urankar was 6th at the regional last year.
• Four of the top 20 Open finishers worldwide and six of the top 43 are competing in this region this weekend.

Thank goodness for the rule allowing extra spots for former champions, because limiting this region to only three spots would be damn near inhumane. Five doesn't even seem quite enough, frankly. Eight competitors from this region were in the top 31 of the worldwide regional standings last year. There are some other great athletes competing this weekend (Ben Smith, Nate Schrader, Lucas Parker, Christy Phillips, etc.), but you can't deny the strength of the Central East men's region as a whole.

The competitiveness of this region also forced me into making some adjustments to my modeling this week. For background on the process of generating these projections, please see my prior post.

Change one this week was the fact that I decided I could not treat Froning like the other competitors. Realistically, the only way he's not making the Games is due to injury. My solution was to give him an 80% chance of finishing top 10 worldwide and a 20% chance of finishing 11-25 worldwide. This leaves open the chance that he could get beat, but only in a scenario where several other guys put up crazy numbers.

Change two was to adjust all my projections to give more of an advantage to the top competitors. My solution was to do the following:

• For each category, look at last year's results and calculate how often athletes in that category finished in the top 25 worldwide. Last week, I calculated how often they finished top 50.
• Additionally, calculate how often they finished between 26-75 worldwide. Last week, I calculated how often they finished 51-100.

The reason for making this change was that in really competitive regions, too many athletes were getting put into that top 50 range. From there, it was just a crapshoot to see who drew better random numbers and qualified for the Games. Now, it's less common for athletes to get into that top 25 category, so doing so gives you a big advantage. For instance, athletes in the top category (top 15 at Games last year) have an 83% shot at finishing top 50 and a 65% shot at finishing top 25. The next highest category (16-50 at Games last year, top 40 worldwide in Regionals last year) had an 81% chance at finishing top 50 but only a 41% shot at finishing top 25. This gives us more separation for the top athletes.

Still, the Central East has five men who all qualified for my top category. The fact is, it's tough to separate them: who would you pick not to make it from those five? But then again, you also have Nick Urankar, who is really solid (26th worldwide in the Regionals last year), and there are others who could legitimately slip into the top 5. As a result, my projections don't look that great for some men that you'd generally consider to be locks for the Games. I'm not sure that's necessarily wrong.

One final note: I made the assumption that if Froning or Holmberg is in the top four, the region gets four spots. If both are in the top five, the region gets five spots. This means if Froning is 5th and Holmberg is 4th, they both still get in. Technically, this isn't how the rules are laid out, but after HQ invited 4th place Kristan Clever from SoCal, I think it's a safe bet that they'd do the same for these guys.

Before I move onto this week's picks, let's look at how last week's picks turned out. Overall, I think pretty well:

• Of the 12 qualifiers from regions I projected, 4 were given more than a 50% shot at making it. There were also 3 athletes who I gave more than a 50% shot who did not make it.
• According to my picks, the longest shots to qualify were Alex Nettey at 16%, Zach Forrest at 18% and Matt Hathcock at 18%.
• If we look at the mean square error (MSE) for my picks compared to some other naive estimates, they did pretty well. 
• The MSE for my picks was 0.045.
• If you had given everyone in each region an equal shot, your MSE would have been 0.064.
• If you had given a 100% chance to the top 3 in the Open in each region, your MSE would have been 0.078.
• If you had given a 100% chance to my top 3 in each region, your MSE would have been 0.057.

So overall, a decent showing. As mentioned above, I made some adjustments, so we'll have to see how they turn out.

As I did last week, I'm limiting these picks to the four regionals that will be broadcast. In this case, that's the Central East men and women and the Mid Atlantic men and women. Without further ado, here are the athletes with the best shot at qualifying from those regions this weekend:

Enjoy the weekend, everyone!