{"id":427,"date":"2021-01-12T23:39:28","date_gmt":"2021-01-12T23:39:28","guid":{"rendered":"https:\/\/bootstrap-it.com\/blog\/?p=427"},"modified":"2021-01-12T23:39:28","modified_gmt":"2021-01-12T23:39:28","slug":"so-malcolm-gladwell-got-the-data-all-wrong-or-did-he","status":"publish","type":"post","link":"https:\/\/bootstrap-it.com\/blog\/?p=427","title":{"rendered":"So Malcolm Gladwell got the data all wrong…or did he?"},"content":{"rendered":"
<\/a><\/a><\/a><\/a><\/a><\/div>\n

In this article, I’ll share some newbie explorations I’ve made in the areas of data analytics and pro hockey.<\/p>\n\n\n\n

I recently embarked on a crazy journey into the world of data analytics. There’s nothing all that crazy about data analytics, mind you. It’s my journey that’s a bit odd.<\/p>\n\n\n\n

You see, I’ve built myself a nice career in cloud and Linux administration<\/a>, but I’m no developer. And, besides some obvious overlap, data is a whole universe apart from administration – a universe where programming on some level just can’t be avoided.<\/p>\n\n\n\n

But parts of my work require me to closely follow the big developing trends in technology<\/a>. And data is big. For years I’ve watched all the (un)cool kids playing around with the numbers that make the modern world work and, frankly, I’m jealous.<\/p>\n\n\n\n

So here I go. I’m going to fumble my way through some very unfamiliar territory, make some dumb mistakes, and have fun. Want to join me?<\/p>\n\n\n\n

This article won’t start with the absolute basic<\/em> basics. If you’re still looking to take your first steps in Python, check this out<\/a>. And if you want to know how to get started with a programming environment like the Jupyter notebooks I use, look over here<\/a>. I’ll assume you’re already comfortable with all that.<\/p>\n\n\n\n

Do Birthdays Matter in Sports?<\/h2>\n\n\n\n

I’ll begin with the question I’m going to try to answer:<\/p>\n\n\n\n

Are you more likely to succeed as an elite athlete if your birthday happens to fall early in the calendar year?<\/p><\/blockquote>\n\n\n\n

It’s been claimed that youth sports that divide participants by age and set the yearly cut off at December 31 unwittingly make it harder for second-half-of-the-year players to succeed. That’s because they’ll be competing against players who are many months older.<\/p>\n\n\n\n

At younger ages, those months can make a very big difference in physical strength, size, and coordination. If you were a minor league coach looking to invest in talent for a better team in a stronger league, who would you choose? And who would benefit over the long term from your extra attention?<\/p>\n\n\n\n

This is where the well-known writer, thinker, (and fellow Canadian) Malcolm Gladwell comes in. Gladwell wasn’t actually the original source of this insight, although he’s the one most often associated with it.<\/p>\n\n\n\n

Rather, those honors fall to the psychologist Roger Barnesly who noticed an oddly distributed birthdate pattern among players at an elite junior hockey game he was attending. Why were so many of those talented athletes born early in the year? Gladwell just mentioned Barnesly’s insight in his book Outliers<\/em>, which was where I came across it.<\/p>\n\n\n\n

But is all that true? Was Barnesly’s observation just an intriguing guess, or does real-world data bear him out?<\/p>\n\n\n\n

Where Does the NHL Hide Its Data?<\/h2>\n\n\n\n

A couple of my kids are still teenagers so, for better or for worse, there’s no escaping the long shadow of hockey fandom in my house. To feed their bottomless appetites for such things, I discovered the existence of a robust official but undocumented API maintained by the National Hockey League. This URL:<\/p>\n\n\n\n

https://statsapi.web.nhl.com\/api\/v1\/teams\/15\/roster\n<\/code><\/pre>\n\n\n\n
…for instance, will produce a JSON-formatted dataset containing the official current roster of the Washington Capitals. Changing that 15<\/code> in the URL to, say, 10<\/code>, would give you the same information about the Toronto Maple Leafs.<\/p>\n\n\n\n
There are many, many such endpoints as part of the API. Many of those endpoints can, in addition, be modified using URL expansion syntax.<\/p>\n\n\n\n
Fun fact: if you look at the site icon in the browser tab while on an NHL API-generated web page, you’ll see the Major League Baseball<\/strong> trademark. How did that<\/strong><\/em> happen?<\/p><\/blockquote>\n\n\n\n
How to Use Python to Scrape NHL Statistics<\/h2>\n\n\n\n
Knowing all that, I could scrape the endpoint for each team’s roster for each player’s ID number, and then use those IDs to query each player’s unique endpoint and read his birthdate. I could then extract the birth month from each NHL player into a Pandas DataFrame where the entire set could be computed and displayed as a histogram.<\/p>\n\n\n\n
Here’s the code I wrote to make all that happen. I’m not going to discuss it in detail here, although that might happen sometime later.<\/p>\n\n\n\n
import pandas as pd\nimport requests\nimport json\nimport matplotlib.pyplot as plt\nimport numpy as np\n\ndf3 = pd.DataFrame(columns=['months'])\nfor team_id in range(1, 11, 1):\n    url = 'https:\/\/statsapi.web.nhl.com\/api\/v1\/teams\/{}\/roster'.format(team_id)\n    r = requests.get(url)\n    roster_data = r.json()\n    df = pd.json_normalize(roster_data['roster'])\n    for index, row in df.iterrows():\n        newrow = row['person.id']\n        url = 'https:\/\/statsapi.web.nhl.com\/api\/v1\/people\/{}'.format(newrow)\n        newerdata = requests.get(url)\n        player_stats = newerdata.json()\n        birthday = (player_stats['people'][0]['birthDate'])\n        newmonth = int(birthday.split('-')[1])\n        df3 = df3.append({'months': newmonth}, ignore_index=True)\ndf3.months.hist()\n<\/code><\/pre>\n\n\n\n
Before moving on, I should add a few notes:<\/p>\n\n\n\n