Monday, August 10, 2015

Assessing Potential Bias

What states/regions have outperformed or underperformed compared to my projections the last couple of years?

First, for a valid look into this topic, you have to consider that there are some teams that simply run very well (or very poorly) at NXN. To this end, I took out any outliers by comparing how teams scored vs. projections from both myself and Meylan. If both Meylan and I were off (in either direction) by 25% or more, I considered those as outliers. That ended up with 14 outliers which you can see at the bottom of this post. Of the remaining 74 teams, I used a simple criteria of average score vs. average projected score.

Quick summary: There was one region that I have apparently overestimated (the South), and I probably underestimated the California girls in 2013. Two other regions were harder to read, and I think the simplest explanation is that both Meylan and I have underestimated one of those regions (the Heartland), and that the Northeast we have both overestimated simply because of a couple teams running a little off and skewing the data. As for the Heartland, I guess it is also possible that nearly half of those teams have simply had big days (Wayzata girls 2013 and boys 2014, and Edina boys both years) since the other five teams all ran pretty close to expectations, but when almost half of the teams fall in the category of “outliers” something (in this case, our projections) is almost surely off. The Northeast is a harder read because most of the region has been very close to projections, but a couple of states (NH and PA) have been missing the mark for whatever reason.

 

Anyway, here is a pair of charts. The first one deals with average scores WITHOUT the outliers included, while the second one deals with ALL teams regardless of whether or not they appear to be outliers. Red means potentially underrated, and yellow means potentially overrated.

 

Non-Outliers Only:

Actual

Watchout

Meylan

Watchout

Meylan

#

282.10

297.80

287.30

CA

105.6%

101.8%

10

348.60

359.00

352.80

HL

103.0%

101.2%

5

260.78

263.78

254.11

MW

101.2%

97.4%

9

261.72

240.72

245.67

NE

92.0%

93.9%

18

326.38

312.25

294.38

NW

95.7%

90.2%

8

318.50

324.88

312.25

SE

102.0%

98.0%

8

373.63

330.13

404.38

SO

88.4%

108.2%

8

253.00

255.38

251.25

SW

100.9%

99.3%

8

 

Outliers Included:

Actual

Watchout

Meylan

Watchout

Meylan

#

281.55

287.73

275.27

CA

102.2%

97.8%

11

281.67

349.33

334.33

HL

124.0%

118.7%

9

280.50

273.60

264.80

MW

97.5%

94.4%

10

271.43

234.48

237.67

NE

86.4%

87.6%

21

309.89

306.22

288.56

NW

98.8%

93.1%

9

318.50

324.88

312.25

SE

102.0%

98.0%

8

392.56

329.22

406.00

SO

83.9%

103.4%

9

256.55

259.73

258.91

SW

101.2%

100.9%

11

 

 

And here is a chart including all the teams. The first number is their actual score, the second is my projection, and the third is Meylan’s. Boys on the left and Girls on the right.

2014:

111

Fayetteville-Manlius NY

171

139

70

Fayetteville-Manlius NY

82

145

159

Wayzata MN

305

260

149

Great Oak CA

217

180

178

North Central WA

258

242

173

Carmel IN

206

179

191

Liverpool NY

243

225

198

Naperville North IL

237

216

195

American Fork UT

160

144

199

Desert Vista AZ

314

330

200

Davis UT

202

193

240

Wayzata MN

256

242

224

Ventura CA

230

263

260

Blacksburg VA

259

230

251

St. Anthony's NY

229

180

272

Lewisville Hebron TX

282

375

262

Edina MN

417

418

276

La Salle Academy RI

332

308

269

Jurupa Hills CA

265

289

278

Saratoga Springs NY

201

225

271

Christian Brothers NJ

234

240

294

Saugus CA

256

310

285

Timpanogos UT

286

315

314

Palatine IL

256

220

299

Severna Park MD

332

340

337

Camas WA

341

295

302

Daniel Boone TN

314

350

342

Willmar MN

355

390

328

Sandburg IL

288

237

347

Davis UT

350

335

363

Carmel IN

301

330

360

Coe-Brown Academy NH

296

322

368

Summit OR

322

331

372

Unionville PA

266

302

372

La Salle Academy RI

205

179

380

Shenendehowa NY

300

323

384

Southlake Carroll TX

313

394

383

Green Hope NC

291

336

399

Brea Olinda CA

322

400

390

Coeur d'Alene ID

364

375

409

Sioux Falls Lincoln SD

392

392

427

Lewisville Marcus TX

314

389

544

The Woodlands TX

322

419

442

American Fork UT

287

295

 

2013:

111

Gig Harbor WA

173

140

108

Wayzata MN

221

202

139

Christian Brothers NJ

172

139

120

Fayetteville-Manlius NY

128

135

174

Brentwood TN

234

230

157

Davis UT

213

213

216

Fayetteville-Manlius NY

139

145

182

Carmel IN

203

200

225

Carmel IN

276

288

211

Monarch CO

215

232

228

St. Xavier OH

251

262

212

Fort Collins CO

205

212

230

American Fork UT

226

199

214

Unionville PA

197

201

231

North Central WA

240

219

276

Great Oak CA

187

155

255

Arcadia CA

234

190

277

Simi Valley CA

362

285

263

Edina MN

406

365

278

Bozeman MT

278

238

299

Northport NY

271

289

279

Pennsbury PA

222

271

301

WC Henderson PA

274

287

282

Davis Senior CA

368

332

303

Severna Park MD

338

314

336

Naperville North IL

356

355

308

Southlake Carroll TX

269

385

352

New Braunfels TX

333

372

322

Wayzata MN

377

341

373

Saratoga Springs NY

370

365

325

Madera South CA

363

298

376

Assumption KY

389

327

344

Davis UT

399

380

398

Southlake Carroll TX

315

385

347

Dana Hills CA

361

326

426

Coe-Brown Academy NH

345

326

375

Central Catholic OR

346

337

430

East Ridge MN

415

399

401

Liverpool NY

247

245

433

The Woodlands TX

372

425

415

Lewisville Hebron TX

443

510

451

Blacksburg VA

442

371

458

Hinsdale Central IL

362

361

521

Bellarmine Prep WA

434

420

 

Team Outliers the last 2 years at NXN =

 

2014 Boys

Exceeded (3): Wayzata MN, North Central WA, Edina MN

Fell short (2): La Salle Academy RI, The Woodlands TX

 

2014 Girls

Exceeded (1): Desert Vista AZ

Fell short (1): American Fork UT

 

2013 Boys

Exceeded (1): Edina MN

Fell short (3): Fayetteville-Manlius NY, Liverpool NY, Hinsdale Central IL

 

2013 Girls

Exceeded (2): Wayzata MN, Davis UT

Fell short (1): Great Oak CA

 

 

4 comments:

  1. This is a great. Wouldn't this be a much richer analysis if you did the same thing, not on team score, but on each individual team runner's final speed rating (or place) relative to your's and Meylan's predictions? This could help to separate out those teams where you prognosticators truly missed the mark with your predictions from those teams who truly underperformed due to one or more runners having a bad day and killing the team score. Your method right not does not discriminate between those two scenarios.

    Second, notice that I only talk about underperforming relative to ability. This is because, by definition, a runner can't overperform (i.e. you can't do better than your ability, although you can "peak" of course for an important race, which would look like a slight "overperformance"). For your analysis, I think you should incorporate this concept. In a measured athletic endeavor, the distribution based on one's ability is not a bell curve! So there should be no outliers above, only below (bad race). Any outliers above are by definition (almost) underprediction.

    Love your work. Please don't take this as a criticism. I just would love to see the outcome of your analysis above with these two concepts included.

    ReplyDelete
    Replies
    1. I did something like that in 2013 in my NXN recap (http://www.dyestat.com/gprofile.php?mgroup_id=44531&do=news&news_id=198252), although not quite at that level (I was just taking out highlights to help tell the story).

      Though I would agree that performance distribution isn't a bell curve, I would disagree that you can't overperform, because this perception is based on observation and projection based on what the athletes have done before, not what they are capable of (which no one really knows). It is absolutely possible to exceed expectations, which is what those outliers are discussing: you run better than you did in previous races (either your best race, or better than your average race).

      I think that those outliers that were exceeding expectations fall into three groups:

      1. Those that were underrated going into the meet because we didn't have as good of a read on them

      2. Those that had setbacks in previous races (off races, injured or sick athletes, etc.) that lowered our expectations

      3. Teams that ran at or near their best moreso than the rest of the field (on average).

      Delete
    2. Oh, and I also posted a shorter review after NXN last year on the message boards: http://www.runnerspace.com/forum/index.php?showtopic=1130392

      Delete
    3. Thanks for your replies. Sorry if I wasn't clear about a point I was trying to make. I'm talking about ability, not expectation or prediction. As a result, I would not treat outliers above and below the same in your analysis, and again, I'm referring to invidual runners, not teams.

      You can overperform relative to expectation, or history, but by definition you can't overperform relative to ability. There's a difference and it suggests that we should treat outliers above one's speed rating different than outliers below a speed rating. If you performed better than you ever did before, this new performance is your ability. The question, perhaps, is how much is a speed rating (composite) a measure of one's ability? I realize of course that it is a measure of his prior performances relative to other runners.

      I think the issue is that running better than a prediction is qualitatively not the opposite of running worse than a prediction. While healing from a prior injury or sickness, deliberately holding back, or good peaking can vault a runner well above their composite speed ratings, what other reasons could there be why a runner would exceed his composite speed rating if that speed rating is a measure of his ability? Conversely, there are millions of reasons why you can run worse than your speed rating.

      I realize a speed rating is not necessarily a measure of one's ability, but I think it would add an interesting dimension to your analysis if you treated it that way, by handling outliers above and below differently.

      Delete