StatOfMind

Does home-court advantage in the NBA matter?

Sun, 06 Dec 2020 00:00:00 +0000

Home-court advantage in the NBA

For obvious reasons, 2020 has not been the best of years, and alongside the terrible societal and economical turmoil created by the COVID-19 pandemic, the sports and entertainment industry has also undergone some dramatic transformations. Of all major sports leagues, the NBA was able to adapt quickly and efficiently to the crisis (albeit at a cost for players and staff, who had to live in a bubble for an extended amount of time), thereby setting a standard to all other leagues on how to safely resume play. Due to the COVID-19 crisis, the NBA paused its season in mid-March 2020, but after extensive planning, was able to resume play in July by creating a bubble from which very few people were able to come in and out. In this bubble, all eligible teams had to complete the remainder of the season and playoffs, and all games tooks place on neutral grounds in which the only fans included the players, coaching and refereeing staff and a few select members of the media.

Now I would pick an arena packed of loud fans any day of the week, but the NBA bubble provided the environment for a unique real-world experiment to estimate the potential impact of fans and home-court advantage on team performance. While the first half of the season saw the schedule operate as normal with home and away games for each team, the second half of the season saw all all games being played on neutral grounds. In this post, we will analyze some NBA game data for the last few seasons to asess whether home-court advantage does indeed impact team performance.

Collecting NBA game data

To begin, I wrote a quick chunk of Python code to scrape NBA game results from the www.basketball-reference.com. The code snippet below iterates through all combinations of year/month since 2015 and extracts the results in a Pandas Dataframe.

import calendar
import itertools
import urllib

import pandas as pd


# define range of years that we are interested in
years = [x for x in range(2015, 2021)]

# Get human-readable names for all months in calendar year
months = [
  x.lower() for x in calendar.month_name if x != ''
]

# generate all year-month combinations
year_month_pairs = list(itertools.product(years, months))
print(year_month_pairs)

# collect all game data sicne 2015
raw_data = dict()
for year_month in year_month_pairs:
    url = f'https://www.basketball-reference.com/leagues/NBA_{year_month[0]}_games-{year_month[1]}.html'
    try:
        df = pd.read_html(url)
        if len(df) > 0:
            print('Success collecting win-loss data for:')
            print(url)
            raw_data[year_month] = df[0]

    except urllib.error.HTTPError as exception:
        print('!!! Failed collecting win-loss data for !!!')
        print(url)
        print(exception)

Running the chunk of code above will store all the data in a raw_dict dictionnary where keys are tuples in the shape of (year, 'month') (for example, (2015, 'january'), (2020, 'aprild'), etc.) and values are Pandas Dataframes containing results played during the correspong year/month pair. For example, you can access all NBA data for December 2019 by typing the command:

raw_data[(2019, 'december')]


Date    Start (ET)  Visitor/Neutral PTS Home/Neutral    PTS.1   Unnamed: 6  Unnamed: 7  Attend. Notes
0   Sat, Dec 1, 2018    5:00p   Milwaukee Bucks 134 New York Knicks 136 Box Score   OT  19812   NaN
1   Sat, Dec 1, 2018    7:00p   Golden State Warriors   102 Detroit Pistons 111 Box Score   NaN 20332   NaN
2   Sat, Dec 1, 2018    7:00p   Brooklyn Nets   88  Washington Wizards  102 Box Score   NaN 15448   NaN
3   Sat, Dec 1, 2018    8:00p   Toronto Raptors 106 Cleveland Cavaliers 95  Box Score   NaN 19432   NaN
4   Sat, Dec 1, 2018    8:00p   Chicago Bulls   105 Houston Rockets 121 Box Score   NaN 18055   NaN
... ... ... ... ... ... ... ... ... ... ...
214 Mon, Dec 31, 2018   7:00p   Memphis Grizzlies   101 Houston Rockets 113 Box Score   NaN 18055   NaN
215 Mon, Dec 31, 2018   7:00p   Boston Celtics  111 San Antonio Spurs   120 Box Score   NaN 18354   NaN
216 Mon, Dec 31, 2018   8:00p   Minnesota Timberwolves  114 New Orleans Pelicans    123 Box Score   NaN 14904   NaN
217 Mon, Dec 31, 2018   8:00p   Dallas Mavericks    102 Oklahoma City Thunder   122 Box Score   NaN 18203   NaN
218 Mon, Dec 31, 2018   9:00p   Golden State Warriors   132 Phoenix Suns    109 Box Score   NaN 16906   NaN

Cleaning NBA game data

Once we have collected the data that we need for this analysis, we can engage in some minor data cleaning/processing in order to prepare the data for further analysis. The chunk of code below concatenates all monthly NBA data into a single DataFrame and cleans up some of the row values and column names. We also assign an additional column score_diff, which shows the score difference between the home and away team. If the score_diff value is positive, then this means that the home team scored more points than the away team (i.e. the home team won). Inversely, if the score_diff value is negative, then this means that the home team scored less points than the away team (i.e. the home team lost).

# concatenate all data into a single DataFrame and clean up entries/columns
winloss_data = (
    pd.concat(data)
    .reset_index()
    .query('Notes!="Playoffs"') # rows with this value only contained null values
    .rename(columns={
        'level_0': 'year',
        'level_1': 'month',
        'Visitor/Neutral': 'visiting_team',
        'PTS': 'visiting_team_score',
        'Home/Neutral': 'home_team',
        'PTS.1': 'home_team_score'
    })
    .drop(columns=['level_2'])
    .fillna({
        'home_team_score': 0,
        'visiting_team_score': 0
    })
)

# convert score data to correct data type
winloss_data[['home_team_score', 'visiting_team_score']] = (
    winloss_data[['home_team_score', 'visiting_team_score']].astype(int)
)

# compute score differentials for each game
winloss_data = (
    winloss_data
    .assign(score_diff = winloss_data['home_team_score'] - winloss_data['visiting_team_score'])
)

Running the chunk of code above will store all the data in a winloss_data Dataframe, which should contain the following data:

winloss_data.head()

	year	month	Date	Start (ET)	visiting_team	visiting_team_score	home_team	home_team_score	Unnamed: 6	Unnamed: 7	Attend.	Notes	score_diff
0	2015	january	Thu, Jan 1, 2015	8:00p	Denver Nuggets	101	Chicago Bulls	106	Box Score	nan	21794	nan	5
1	2015	january	Thu, Jan 1, 2015	8:00p	Sacramento Kings	110	Minnesota Timberwolves	107	Box Score	nan	13337	nan	-3
2	2015	january	Fri, Jan 2, 2015	7:00p	Cleveland Cavaliers	91	Charlotte Hornets	87	Box Score	nan	19307	nan	-4
3	2015	january	Fri, Jan 2, 2015	7:00p	Brooklyn Nets	100	Orlando Magic	98	Box Score	nan	17008	nan	-2
4	2015	january	Fri, Jan 2, 2015	7:30p	Dallas Mavericks	119	Boston Celtics	101	Box Score	nan	18624	nan	-18

A quick analysis of home-court advantage in the NBA

Now that we have gathered and cleaned up our NBA game data, we can proceed to a small analysis of the data. To begin, we can simply look at the summary statistics for the score_diff column across different seasons.

winloss_data.groupby('year').score_diff.describe()

year	count	mean	std	min	25%	50%	75%	max
2015	1311	2.39054	13.5141	-54	-7	4	11	53
2016	1316	3.02736	13.5698	-51	-6	4	12	50
2017	1309	3.08327	13.8825	-44	-6	4	12	49
2018	1312	2.36662	13.7199	-48	-7	4	11	61
2019	1319	2.77786	14.3607	-56	-7	4	12	50
2020	1258	1.86248	13.4501	-41	-7	0	10	49

We can immediately see that the 2020 season has different summary statistics values than all other seasons. On average, the “home” team scored ~1.5 fewer points during the 2020 season (remember that score_diff shows the score difference between the home and away team. If the score_diff value is positive, then this means that the home team scored more points than the away team.). Similarly, the median value for the score_diff was equal to 0 during the 2020 season, whereas it was equal to 4 for all other seasons since 2015. This indicates that until the 2020 season, 50% of teams playing at home scored 4 or more points than their opponents. In 2020, 50% of teams playing at home scored the same number of points than their opponents. If we recall that half of the 2020 season was played on neutral grounds, the numbers above indicate (but do not prove) that home-court advantage may have a significant impact on the win probability of the home team.

We can also visualize the overall distribution of the score differentials between the home and away teams across different seasons:

ax = (
    winloss_data
    .hist(
        by='year',
        column = 'score_diff',
        bins=50,
        figsize=(18, 10),
        layout=(2, 3), 
        color='grey',
        rot=0
    )
)

The distribution of the score differentials between the home and away teams also displays some interesting patterns. The distributions for seasons 2015-2019 appear to follow a bi-modal distribution, with two spikes around zero (there is no such thing as a draw in basketball). In all cases, the “positive” spike is larger than the “negative” spike, which aligns with the previous observation that home teams were more likely to win their games during those seasons. This bi-model distribution is not as clearly defined for the 2020 season, where instead we see a significantly larger “negative” spike, which indicates that home teams were more likely to lose their games during that season.

When plotting the normalized cumulative distribution function for the score differentials between the home and away teams, we obtain the following figure:

cum_score_diff = (
    winloss_data
    .groupby(['year', 'score_diff'])
    .size()
    .reset_index()
    .pivot(
        index='score_diff',
        columns='year',
        values=0
    )
    .fillna(0)
    .cumsum()
)

(cum_score_diff / cum_score_diff.max()).plot(figsize=(10, 8))

Again, the 2020 season stands apart from all other previous seasons included in our dataset, which shows a distinct uptick of games on the negative side of the zero mark that denotes games where the “away” team won. However, if we really want to show the potential impact of home-court advantage, we can focus on the before/after distribution of score differentials during the 2020 season. To begin, we define the months during games were played inside and outside of the bubble:

months_before_bubble = ['october', 'november', 'december', 'january', 'february']
months_after_bubble = ['july', 'august', 'september']

We can then compute and plot the normalized cumulative distribution function for the score differentials before and after the bubble:

cum_score_diff_2020 = (
    winloss_data
    .assign(
        is_in_bubble = ['No' if x in months_before else 'Yes' for x in winloss_data.month]
    )
    .query('year==2020')
    .groupby(['is_in_bubble', 'score_diff'])
    .size()
    .reset_index()
    .pivot(
        index='score_diff',
        columns='is_in_bubble',
        values=0
    )
    .fillna(0)
    .cumsum()
)

ax = (cum_score_diff_2020 / cum_score_diff_2020.max()).plot(figsize=(10, 8))
ax.set_xlabel('(Home Team Score) - (Away Team Score)')
ax.set_ylabel('Normalized Cumulative Function')

The figure clearly shows the impact that moving to a “bubble” environnment had on the score differentials between home and away teams. By entering a bubble in which all games were played on neutral arenas, the NBA negated the impact that fans could have on the outcome of games. As a result, we can reasonably conclude that the NBA bubble had a real impact on team performance, and that home-court advantage is a real phenomenom.

Comparing rental prices in the US market

Mon, 11 Dec 2017 00:00:00 +0000

Rental Prices in the US

As a fortunate resident of New York City, I get to enjoy the many activities offered by a city that never really turns off its lights. The price to pay for such entertainment, however, are the crazy rental fees. I have stopped counting the number of conversations I have had with friends about some of the exhorbitant living costs in NYC. I have also drooled over multi-million mansions with sprawling views of the city, and wondered how studios the size of a shoe box could claim such high prices. Fortunately for me, I have been living in the same spot for the last few years, but am now expecting to start looking again soon.

This means that I am starting to look for both a new neighborhood and a new apartment, and for that I have been primarily relying on the Streeteasy and Zillow wesbites. Eventually, keeping track of all this gets a little confusing, and I can only imagine how much worse it gets for people just moving into NYC.

For these reasons, I decided to see if I could leverage some open-source data in order to extract and visualize some information out of it. In the following post, I will collect and analyze some of the great data made available by Zillow, while also going through the steps that allowed me to create the web application displayed below

Obtaining Zillow data & Forecasts

Obtaining the data for this project is very straightforward, and does not require any scraping or cleaning. All the data can be directly downloaded from the following link, and contains monthly median rental prices in neigborhoods of various US cities between 2010 and now. In order to produce forecasts of the time series represented in the Zillow dataset, I leveraged the additive HoltWinters function, which comes as part of R’s core packages). Briefly, the additive Holt Winters model assumes that the value of a time series \(Y\) at time \(t\) can be represented as:

\[y_{t} = b_{1} + b_{2}t + S_{t} + \epsilon_{t}\]

where \(b_{1}\) represents the base signal of the time series, \(_{2}\) denotes the linear trend component, \(S_{t}\) is the additive seasonal factor and \(\epsilon_{t}\) is the random error component of the time series. The Holt Winters additive seasonal model assumes that the amplitude of the seasonal component of the time series is independent of the average level of the series itself. I thought that this assumption fit well with the Zillow data, so decided to work with that. I also assessed other methodologies for time series forecasting, but it eventually turned into a bigger project than I had intended. Instead, I have planned to write a comparative analysis for the various time series libraries available in both Python and R. In the meantime, let’s show how the app was built!

Building the Rental Predictor Shiny application

The next step was to build a Shiny app that allows anyone to look at forecasts and seasonality of different neighborhoods. For simplicity, I have hosted the app on shinyapps.io, but in order to ensure full reproducibility, I have containerized the full application using Docker. If you have not used or heard of Docker, their own website gives a very good description:

Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

With that in mind, the development workflow used to create this Shiny app was:

Build the shiny app locally (i.e. on your local machine) and ensure that everything works just as expected.
Create a file called Dockerfile, in which we specify all the required instructions and commands, environnment variables and configurations.
Build and run the Docker container to check that the app behaves as expected. If not, iterate over the first step until you are satisfied.
Reap the benefits of a fully reproducible app that can be ran on any OS platform!

The source code for the full web app can be found at the following GitHub repository, and has the following directory structure:

.
├── Dockerfile
├── LICENSE
├── README.md
├── app
│   ├── data
│   │   ├── Neighborhood_MedianRentalPrice_1Bedroom.csv
│   │   ├── Neighborhood_MedianRentalPrice_2Bedroom.csv
│   │   ├── Neighborhood_MedianRentalPrice_3Bedroom.csv
│   │   ├── Neighborhood_MedianRentalPrice_4Bedroom.csv
│   │   └── Neighborhood_MedianRentalPrice_Studio.csv
│   ├── server.R
│   ├── static
│   │   └── dygraph.css
│   ├── ui.R
│   └── utils.R
├── download_data.sh
├── rental_predictor.R
└── shiny-server.sh

1 directory, 10 files

By cloning this Github directory and typing the commands below, you will be able to run the web app yourself. You should also make sure that Docker is installed on your machine, which can be achieved by following the instructions in this tutorial.

# build Docker container with tag name rental-predictor
docker build -t rental-predictor .

# run Docker container and mirror port 4000
docker run -p 4000:4000 -t rental-predictor

# Note: the above was tested on a Ubuntu machine hosted on DigitalOcean.
# For convenience, I used their Docker One-Click Applications,
# which automatically installs Docker during the booting process.

Et voila, you are done! A snapshot of the app is displayed below, or you can visit the app at the following link. It allows to select a city and neighborhood, at which point the plot described in the previous section are surfaced to the user:

NFL Series

Sun, 08 Oct 2017 00:00:00 +0000

If you have previously attempted to analyze NFL data, it is likely that you have tried to scrape ESPN or football-reference, which provides a wealth on statistics surrounding game data. However, if you ever wanted to obtain truly in-depth data, then it is likely that you found yourself leveraging the API maintained by NFL.com. Unfortunately, it’s data is surfaced in a JSON format that leaves a lot to be desired (i.e. it’s a nightmare). Lucky for me, I was recently scrolling through my Twitter feed and came across an interesting mention of a new R package called nflscrapR. After some exploration, I came through this quote from the author of the package:

NFL fans and sports enthusiastic alike, I would like to introduce the nflscrapR package, an American football data aggregator that will scrape, clean, and parse play-by-play data across games, seasons, and careers.

The nflscrapR essentiallys surfaces all play-by-play data for the last 7 seasons, and this has motivated me to start a deep dive on NFL data. I plan to have two main topics, one that focusses on players at specific positions, and another focussing on team dynamics and patterns. To kick this series off, we will begin with an exploration into the performance of NFL’s best running backs.

1. Prerequisites

In order to reproduce the figures below, wou will need to have R (v3.2.3 or above) installed on your machine. There are also a couple of additional libraries that will be required. Details on how to install those are shown in the commands below.

# install.packages('devtools')
library(devtools)

#> Skipping install for github remote, the SHA1 (05815ef8) has not changed since last install.
#>   Use `force = TRUE` to force installation

devtools::install_github(repo = "maksimhorowitz/nflscrapR", force=TRUE)

2 Collecting the Data

With the nflscrapR library now installed, you are now ready to collect play-by-play data for the 2016-2017 NFL season. Start by loading the library and collect the data using the command below:

# Load the package
library(nflscrapR)
library(ggplot2)
library(dplyr)
library(pheatmap)

# Collect data for 2016 NFL season
pbp_2016 <- season_play_by_play(2016)

Overall the pbp_2016 dataframe contains 100 data points for 45,737 plays, but for the purpose of this post, we will be focussing exclusively on fields related to running backs (In future posts, we will explore data relevant to other positions on the football field). In addition, we’ll focus primarily on frequently used running backs, which we empirically define as any player that has had at least 200 rushes over the course of the 2016-2017 season.

# Get all players with at least 200 rushes during the season
min_rush_cnt <- 200
rush_cnt <- pbp_2016 %>% filter(PlayType == 'Run') %>%
                         group_by(Rusher) %>% 
                         summarise(rush_cnt = n(),
                                   total_yards = sum(Yards.Gained),
                                   mean_yards = round(mean(Yards.Gained), 2)) %>%
                         filter(rush_cnt >= min_rush_cnt) %>%
                         arrange(desc(rush_cnt))

# Get all rushing data for eligible players
rushing_stats <- pbp_2016 %>%
                 filter(PlayType == 'Run' & Rusher %in% rush_cnt$Rusher & Yards.Gained <=50) %>%
                 filter(down!=4 & !is.na(down)) %>%
                 filter(!is.na(RunLocation))

Altogether, we find that a total of 19 players rushed over 200 times during the 2016-2017 season. A short summary of their performance is show below.

3. Who are the most consistent and productive running backs?

When talking about the overall performance of running backs, it is common for people to report the total number of yards that were rushed for, or the average yards per run. While these are perfectly acceptable numbers to share, I’ve always felt like they did not tell the whole story. For example, a player could have a high average yards per run, only for us to realize that he actually often loses yards on a run but makes up for it with a few very long runs. Therefore, I started by looking at the overall distribution of number of yards gained/lost for each play, with the hope that this would reveal whether some players were more consistent on a play-by-play basis than others. We can use the ggplot2 library to generate a density plot of yards gained per play for each of our eligible players:

# Compare distribution of rushes for eligible players
ggplot(rushing_stats, aes(x = Yards.Gained, y = Rusher, fill=Rusher)) +
       geom_joy(scale = 3) +
       theme_joy() +
       scale_fill_manual(values=rep(c('gray', 'lightblue'), length(rushing_stats$Rusher)/2)) +
       scale_y_discrete(expand = c(0.01, 0)) +
       scale_x_continuous(expand = c(0, 0)) +
       theme(legend.position="none") +
       labs(x="Yards gained per play" ,y="")

Overall, we see that most running backs have a similar distribution of yards gained by run. However, we can see that LeSean McCoy (7th distribution from the top) has a much “flatter” distribution (i.e. more variance in the distribution of yards gained per run), meaning his performance can be a lot more variable/unpredictable for any given run.

4. When are running backs used?

Another statement that is also commonly reported is that running backs are primarily used in early downs. To verify whether this is generall true, we can compute the total amount of runs that each player made across different downs, and go even further by breaking this down by quarter too. The code chunk below counts the number of runs that each rushing back made during pairs of downs and quarters.

# Compare when rushers are used
usage_stats <- pbp_2016 %>% filter(!is.na(down) & Rusher %in% rush_cnt$Rusher & qtr!=5) %>%
                             group_by(Rusher, down, qtr) %>%
                             summarise(cnt = n()) %>%
                             mutate(qtr_down = paste("Q", qtr, "- Down: ", down, sep=""))

We can then leverage the d3heatmap to quickly generate a simple heatmap of how often running backs are used during specific downs and quarters.

library(d3heatmap)

# pivot dataframe
usage <- usage_stats %>% dcast(Rusher ~ qtr_down, value.var = "cnt")

# clean data
row.names(usage) <- usage$Rusher
usage <- usage %>% select(-Rusher)
usage[is.na(usage)] <- 0

# normalize data
usage_norm <- t(apply(usage, 1, function(x) x/sum(x)))

# plot heatmap of proportions of rushes by different field locations and gaps
p <- d3heatmap(usage_norm,
               colors="Blues",
               Colv=FALSE,
               show_grid=3)

saveWidget(p, file="rusher_usage_down_quarter.html")

Proportion of rushes per quarter and downs for NFLs best running backs

In the plot above, we are essentially plotting the usage of each running back as a function of what stage of the game we are in. As we can see, it is abundantly clear that running backs are primarily used in the first two downs, and rarely during the third and fourth downs. Overall, there does not appear to be significant differences between how players are used. However, it does not answer whether some running backs perform better on certains downs, which is what we will address now.

5. Are some running backs better on certain downs?

Another question we can ask ourselves is whether some running backs perform better on later downs. To visualize this data, we can again generate a density plot of yards gained per play for each of our eligible players, while also facetting the data by downs.

# Compare distribution of rushes by downs
ggplot(rushing_stats, aes(x = Yards.Gained, y = down)) +
       geom_joy(scale=1, rel_min_height=.03, fill='black') +
       scale_y_discrete(expand = c(0.01, 0)) +
       xlab('Value') +
       facet_wrap(~Rusher, scales='free', ncol=3) +
       theme_joy() +
       theme(axis.title.y = element_blank())+
       labs(x="Yards gained per play" ,y="Down")

Again, we do not see any striking differences between players and the distribution of yards gained by down. However, it is interesting to note that most “long runs” (10 yards or above) tend to occur on the first down. When we look closely, we can also see that some rushers such as DeMarco Murray do exhibit visual differences between yards gained by downs. In this case, the “mass” of yards gained on the third down is much closer to zero than when compared to the “mass” for the first and second downs, which suggests that he may struggle during this down (this could be attributable to many factors: stamina, weaker offensive line on 3rd downs, etc…)

6. Where do the best running backs run?

It is fairly well accepted that the performance of a running back will be heavily influenced by the strength of the offensive line in front of them. With that in mind, let’s start by looking at the field location in which different running backs prefer to run. The plot below shows the number of yards gained by each running back based on which side of the field they ran towards (left, middle or right).

ggplot(data=rushing_stats, aes(x=RunLocation, y=Yards.Gained, color=RunLocation)) +
       geom_jitter(position=position_jitter(0.2)) +
       stat_summary(fun.data=mean_sdl, mult=1, 
                    geom="pointrange", color="black") +
       scale_color_brewer(palette="Dark2") + theme_minimal() +
       facet_wrap(~Rusher)

We can take this further by looking at the field location in which different running backs prefer to run. This can be achieved by generating a matrix that contains the proportion of rushes by field location for each player.

# Get proportions of rushes on different field locations
rush_locations <- rushing_stats %>% filter(PlayType=='Run') %>%
                                    filter(!is.na(RunLocation)) %>%
                                    group_by(Rusher, RunLocation) %>%
                                    summarise(rush_cnt = n()) %>%
                                    mutate(freq = rush_cnt / sum(rush_cnt))

loc_mat <- rush_locations %>% dcast(Rusher ~ RunLocation, value.var = "freq")
row.names(loc_mat) <- loc_mat$Rusher
loc_mat <- loc_mat %>% select(-Rusher)

The content of the loc_mat matrix contains the preferred rush locations of each running back, and can be plotted as a clustered heatmaps using the pheatmap library in R.

# Plot heatmap of proportions of rushes by different field locations
pheatmap(loc_mat, border="white", color = brewer.pal(9,"Blues"), cluster_cols=FALSE)

The plot above highlights which running back are most similar in their run locations. Rushers such as J. Ajayi, E. Elliot and M. Ingram, clearly avoid running in the middle of the field. On the flipside, rushers such as M. Gordon D. Johnson, S.ware and F. Gore clearly prefer running down the middle rather than to the sides. These patterns could be attributed to the running styles of each rushers (speed, mobility, strength etc…), but also the strength of the offensive line at particular positions.

7. Who creates the gaps for the running backs?

We can also explore the number of yards gained by each running back based on the offensive line positions that created space for them.

ggplot(data=rushing_stats %>% filter(!is.na(RunGap)), aes(x=RunGap, y=Yards.Gained, color=RunGap)) +
       geom_jitter(position=position_jitter(0.2)) +
       stat_summary(fun.data=mean_sdl, mult=1, 
                    geom="pointrange", color="black") +
       scale_color_brewer(palette="Dark2") + theme_minimal() +
       facet_wrap(~Rusher)

The proportions of run opportunities that was enabled by each offensive line position can also be summarized in a matrix using the command below.

# Get proportions of gaps created by different offensive line positions
rush_gaps <- rushing_stats %>% filter(!is.na(RunGap)) %>%
                               filter(!is.na(RunGap)) %>%
                               group_by(Rusher, RunGap) %>%
                               summarise(rush_cnt = n()) %>%
                               mutate(freq = rush_cnt / sum(rush_cnt))

gap_mat <- rush_gaps %>% dcast(Rusher ~ RunGap, value.var = "freq")
row.names(gap_mat) <- gap_mat$Rusher
gap_mat <- gap_mat %>% select(-Rusher)

# Plot heatmap of proportions of rushes by different field gaps
pheatmap(gap_mat, border="white", color = brewer.pal(9,"Blues"), cluster_cols=FALSE)

Again, we see many differences among the leagues top running backs. Unsurprisingly, a number of players have the most run opportunities created by the guard position, but a few players such as F. Gore, L. McCoy and D. Johnson run in gaps created by the tackle position. Finally, S. Ware from the Kansas City Chiefs often runs through gaps created by tight ends, which may be more representative of the team’s formation.

Conclusion

In this introductory post, we have explored the performance and patterns of some of NFL’s best running backs. Overall, it was a fairly superficial analysis, as it never considered interactions with other components of the team, or temporal patterns, but it does show the depth and power of this data. In the next series, I will dive into the performance and behavior of wide receivers, so stay tuned!

Building a Kafka and Spark Streaming pipeline - Part I

Sun, 25 Sep 2016 00:00:00 +0000

Many companies across a multitude of industries are currently maintaining data pipelines used to ingest and analyze large data streams. In effect, the proper implementation of such pipelines belongs to the realm of “data engineering”, and represents a gateway to interesting data science-related problems. Traditional machine learning methods have been developed to work using batch or offline approaches, but there are fewer options when we start considering solutions for true streaming problems.

In this series of posts, we will build a locally hosted data streaming pipeline to analyze and process data streaming in real-time, and send the processed data to a monitoring dashboard. As the figure below shows, our high-level example of a real-time data pipeline will make use of popular tools including Kafka for message passing, Spark for data processing, and one of the many data storage tools that eventually feeds into internal or external facing products (websites, dashboards etc…)

1. Setting up your environnment

We will assume that you have nothing installed on your machine. To begin, it is useful to check whether you have Java installed or your machine, and if yes, whether it is at version>=1.8.

TLV-private:ThomasVincent$ java -version
java version "1.8.0_51"
Java(TM) SE Runtime Environment (build 1.8.0_51-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.51-b03, mixed mode)

If that is the case, then we can proceed with the next steps, otherwise you might want to install Java using the instructions here: https://java.com/en/download/help/mac_install.xml

2. Zookeeper

2.1. Installing Zookeeper

Zookeeper is an Apache project specifically built with the intention of helping us build and maintain distributed applications. In short, it is an invaluable tool to take much of the heavy lifting out of building distributed processes. Some further explanations and useful links can be found at the following StackOverflow link

To begin, go ahead and download Zookeeper (Release 3.4.9 (stable) ) from this link. Once the tar-zipped file has been downloaded, move it to your working directory, unpack it, and change your working directory to the Zookeeper directory.

TLV-private:ThomasVincent$ mv zookeeper-3.4.9.tar.gz /path_to_working_directory/
TLV-private:ThomasVincent$ tar -zxf zookeeper-3.4.9.tar.gz
TLV-private:ThomasVincent$ cd zookeeper-3.4.9

2.2. Configuring Zookeeper

At this point, you can create a new directory data using the mkdir command.

TLV-private:ThomasVincent$ mkdir data

and also edit the Zookeeper configuration file located in the conf directory with the command

vim conf/zoo.cfg

Note that vim will automatically create the zoo.cfg file if it does not already exist. You will usually find a heavily commented conf/zoo_sample.cfg file in most default installations of Zookeeper, but if not, insert the following 5 lines in the configuration file.

tickTime=2000
dataDir=/path_to_your_working_directory/zookeeper-3.4.9/data
clientPort=2181
initLimit=5
syncLimit=2

2.3. Starting Zookeeper

We are now ready to start our Zookeeper server, which can be achieved by running the zkServer.sh shell script:

TLV-private:ThomasVincent$ bin/zkServer.sh start

After executing this command, you should see the following output:

TLV-private:ThomasVincent$ ZooKeeper JMX enabled by default
Using config: /Users/ThomasVincent/Desktop/StatOfMind/kafka_spark_pipeline/zookeeper-3.4.9/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

You can also launch the Zookeeper CLI, which will allow you to connect to the Zookeeper server:

TLV-private:ThomasVincent$ bin/zkCli.sh

Executing this command should generate a fair amount of output, but you should see the following:

Connecting to localhost:2181
...
...
...
Welcome to ZooKeeper!
...
...
...
WATCHER::
WatchedEvent state:SyncConnected type:None path:null

3. Apache Kafka

Apache Kafka is a high-throughput distributed messaging system in which multiple producers send data to a Kafka cluster and which in turn serves them to consumers. Because of its efficiency and resiliency, it has become one of the de facto tool to consume and publish streaming data, with applications ranging from AdTech, IoT and logging data.

3.1. Installing Kafka

Let’s start by downloading the Kafka binary and installing it on our machine. For this, you will need to get version 2.11, which can be obtained here. Once this is done, simply move the Kafka binary to your working directory (for simplicity, let’s say the one where you placed your ZooKeeper binary) and unzip the Kafka binary.

TLV-private:ThomasVincent$ tar -zxf kafka_2.11-0.10.0.1.tgz
TLV-private:ThomasVincent$ cd kafka_2.11-0.10.0.1

3.2. Starting the Kafka server

Because Kafka depends on Zookeeper to maintain and distribute tasks, we need to start ZooKeeper before starting the Kafka broker.

TLV-private:ThomasVincent$ bin/zookeeper-server-start.sh config/zookeeper.properties
TLV-private:ThomasVincent$ bin/kafka-server-start.sh config/server.properties

You are now ready to start your Kafka server, using the command below

TLV-private:ThomasVincent$ bin/kafka-server-start.sh config/server.properties

Executing this command will generate a large number of logging lines, but the start and end should look like the following:

[2016-09-25 11:26:38,298] INFO starting (kafka.server.KafkaServer)
[2016-09-25 11:26:38,348] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
...
...
...
[2016-09-25 11:26:42,039] INFO [Kafka Server 0], started (kafka.server.KafkaServer)

3.3. Starting your first Kafka topic

Next, you can initialize a Kafka topic by using the kafka-topics.sh utility. In a new terminal window, type the command below to create a new topic called test-topic with a single partition and one replica factor.

TLV-private:ThomasVincent$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test-topic

Created topic "topic-name".
...
...
...
[2016-09-25 11:34:32,245] INFO Partition [test-topic,0] on broker 0: No checkpointed highwatermark is found for partition [test-topic,0] (kafka.cluster.Partition)

We can also list the topics currently in the Kafka server by using the kafka-topics.sh utility script

TLV-private:ThomasVincent$ bin/kafka-topics.sh --list --zookeeper localhost:2181
test-topic

3.4. Producing and consuming messages with Kafka

None of what we have done is very useful if no data is sent to the Kafka brokers. Here, we will configure a producer to send messages to our broker. Configurations for a single producer can be found in config/server.properties. A quick check in this file tells us that our broker listens to localhost:9092. Therefore, we use the kafka-console-producer.sh utility to create a producer to send messages to localhost:9092 under our topic of choice. Once the producer is running, it will wait for input from stdin and publish to the Kafka cluster. The default setting is to have every new line be published as a new message, but tailored producer properties can be specified in the config/producer.properties file. The command below starts a producer and writes a couple of messages to stdin:

TLV-private:ThomasVincent$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test-topic
Hello
This is a test
Hello again

We can then consume those messages using the kafka-console-consumer.sh utility script

TLV-private:ThomasVincent$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic test-topic
Hello
This is a test
Hello again

So our consumer is successfully reading messages from our producer (via the broker). If you continue typing random messages in the producer terminal window, you will see them be printed out in the consumer terminal window.

4. Setting up Spark

So far we have initialized Zookeeper, set up a Kafka cluster, started a producer that sends messages to a Kafka broker, and a a consumer that reads all messages send by the producer. In a real-world setting, this last step would be used to ingest, transform and possibly analyze the incoming data. Tools such as Spark or Storm work are some of the popular options used with Kafka for this type of use-case. In this series, we will leverage Spark Streaming to process incoming data. To begin we can download the Spark binary at the link here (click on option 4) and go ahead and install Spark. Note, it may also be wortwhile to include the following in your .bashrc file so that you do not have to repeat these steps every time you launch a new shell:

export SPARK_HOME=/your_path_to_spark_directory/spark
export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH
PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip:$PYTHONPATH

4. Word count with Kafka and Spark Streaming

In this first part of the series, we will implement a very simplistic word count script (the “Hello World!” equivalent for Spark). However, it also seems vapid to limit ourselves to such an easy example when we have such great technology at our disposal, so the second part of this series will focus on implementing more complicated examples that may be applicable in real life scenarios.

4.1. The most vanilla word count script

First, let’s start by writing our word count script using the Spark Python API (PySpark), which conveniently exposes the Spark programming model to Python. You can copy the chunk of code below into a file called kafka_wordcount.py to be placed in your working directory. While the code is self-explanatory, it is important to note that we are making use of Spark Streaming, a module built on top of Spark Core. Spark Streaming leverages Spark Core’s fast scheduling capability to perform streaming analytics and ingests data in mini-batches while performing RDD transformations on those mini-batches of data.

import sys
from pyspark import SparkContext
from pyspark import SparkConf
from pyspark.streaming import StreamingContext
from pyspark.streaming.kafka import KafkaUtils
from pyspark.sql.context import SQLContext

if __name__ == '__main__':
    if len(sys.argv) != 3:
        print("Usage: kafka_wordcount.py <zk> <topic>", file=sys.stderr)
        exit(-1)

    sc = SparkContext(appName="PythonStreamingKafkaWordCount")
    ssc = StreamingContext(sc, 1)

    zkQuorum, topic = sys.argv[1:]
    kvs = KafkaUtils.createStream(ssc, zkQuorum, "spark-streaming-consumer", {topic: 1})
    lines = kvs.map(lambda x: x[1])
    lines.pprint()

    counts = lines.flatMap(lambda line: line.split(" ")) \
                  .map(lambda word: (word, 1)) \
                  .reduceByKey(lambda a, b: a+b)
    counts.pprint()

    ssc.start()
    ssc.awaitTermination()

You can now open up a new terminal window and from your working directory, input the following command:

TLV-private:ThomasVincent$ bin/spark-submit --jars external/kafka-assembly/target/scala-*/spark-streaming-kafka-assembly-*.jar /Users/ThomasVincent/Desktop/kafka_wordcount.py localhost:2181 test-topic

If you continue typing random sentences in the producer terminal window, you will now see the output of your wordcount script being returned in your consumer terminal window! Congratulations, you have just successfully ran your first Kafka / Spark Streaming pipeline.

4.2. Leveraging DataFrame and SparkSQL

PySpark comes with a number of useful modules that make working with data a whole lot easier. Two particurlaly oft-used modules are SparkSQL and DataFrame, which both provide support for processing structured and semi-structured data. For more information on the many useful features of these two modules, please refer to this link. For now, we can further extend our word count example by integrating the DataFrame and SparkSQL features of Spark.

Copy the chunk of code below into a file called kafka_spark_dataframes.py and place it in your working directory.

import sys
from pyspark import SparkContext
from pyspark import SparkConf
from pyspark.streaming import StreamingContext
from pyspark.streaming.kafka import KafkaUtils
from pyspark.sql import Row, DataFrame, SQLContext


def getSqlContextInstance(sparkContext):
    if ('sqlContextSingletonInstance' not in globals()):
        globals()['sqlContextSingletonInstance'] = SQLContext(sparkContext)
    return globals()['sqlContextSingletonInstance']


# Convert RDDs of the words DStream to DataFrame and run SQL query
def process(time, rdd):
    print("========= %s =========" % str(time))
    try:
        # Get the singleton instance of SparkSession
        sqlContext = getSqlContextInstance(rdd.context)

        # Convert RDD[String] to RDD[Row] to DataFrame
        #rowRdd = rdd.map(lambda w: Row(word=w))
        rowRdd = rdd.map(lambda w: Row(word=w[0], cnt=w[1]))
        #rowRdd.pprint()
        wordsDataFrame = sqlContext.createDataFrame(rowRdd)
        wordsDataFrame.show()

        # Creates a temporary view using the DataFrame.
        wordsDataFrame.createOrReplaceTempView("words")

        # Do word count on table using SQL and print it
        wordCountsDataFrame = \
             spark.sql("select SUM(cnt) as total from words")
        wordCountsDataFrame.show()
    except:
       pass

if __name__ == '__main__':
    if len(sys.argv) != 3:
        print("Usage: kafka_spark_dataframes.py <zk> <topic>", file=sys.stderr)
        exit(-1)

    sc = SparkContext(appName="PythonStreamingKafkaWordCount")
    sqlContext = SQLContext(sc)
    ssc = StreamingContext(sc, 1)


    zkQuorum, topic = sys.argv[1:]
    kvs = KafkaUtils.createStream(ssc, zkQuorum, "spark-streaming-consumer", {topic: 1})
    lines = kvs.map(lambda x: x[1])
    words = lines.flatMap(lambda line: line.split(" ")) \
        .map(lambda word: (word, 1)) \
        .reduceByKey(lambda a, b: a+b)

    words.foreachRDD(process)

    ssc.start()
    ssc.awaitTermination()

In a new terminal window, input the following command (and make sure you stop the previous word count script!):

TLV-private:ThomasVincent$ bin/spark-submit --jars external/kafka-assembly/target/scala-*/spark-streaming-kafka-assembly-*.jar /Users/ThomasVincent/Desktop/kafka_spark_dataframes.py localhost:2181 test-topic

Once again, typing random sentences in the producer terminal window will return a wordcount in your consumer terminal window! The only difference to the previous script is that we are now leveraging the SparkSQL module to perform this task.

Next Steps

As mentioned previously, the word count script represents a basic use of the technology at hand, and does not do justice to the capabilities of the tools. In the next part of this blog series, we will look to implement some more complicated scenarios, with a focus on data science rather than straightforward engineering pipeline.

A map of elevators in NYC

Wed, 31 Aug 2016 00:00:00 +0000

Not too long ago, I came across a random tweet pointing to a GitHub repository full of miscallaneous datasets. I was imemdiately excited by the various data science and visualization problems tha I could take on and immediately decided to get my hands dirty. One of the first datasets that I saw was a dataset called eleavtors. As stated on the github repo, it contains

A list of registered elevator devices in New York City provided by the Department of Buildings in response to a September 2015 FOIL request. The data was received on a DVD in November 2015. The spreadsheet contains information on 76,088 elevators in New York City.

As a resident of NYC, and someone who has mayny times complained abut going to friends houses with no elevators (admittedly a first-world problem…), I wanted to take a look at that data. (Full disclosure, half-way through this little project, I realized that fivethirtyeight had already beaten me to the punch and performed an analysis of its own.

A map of elevators in NYC

The main point of this post is to illustrate how simple it is to produce interactive leaflet map using the leaflet and htmlwidgets package in R. in fact, it is only a single line of code to produce the map above, the rest is just processing the data into the correct format. A word of advice when using the leaflet package at the zipcode level: the library expects 5 digits for all US-based zipcode, while many databases produce zipcodes that sometimes contain 4 digits. If that happens, it is necessary to pad “0”s to the zipcode until the string in question has a length of five. This can be achieved using a neat little trick that is shown in line 11 of the code below!

library(dplyr)
library(data.table)
library(leaflet)
library(htmlwidgets)

# read in prison details and zip coordinates
# load required libraries
library(data.table)
library(rCharts)
library(dplyr)
library(leaflet)

# read in source data and process
elevators_dt <- fread('nyc_elevators.csv')
elevators_dt <- elevators_dt[elevators_dt[['STREET_NAME']] != 'DUMMY RECORD', ]

# define function to extract zip code of elevators
extract_zip <- function(x) {
    if(nchar(x) == 9) {
        return(substring(as.character(x), 1, 5))
    }
    else{
        return(x)
    }
}

# count frequency of elevators by zip code and coordinates
zip_cnt <- elevators_dt[, .N, by=ZIP_CODE]
zip_cnt[, 'zip' := unlist(sapply(zip_cnt$ZIP_CODE, function(x) extract_zip(x)))]
coord_pairs <- elevators_dt[, .N, by=list(LATITUDE, LONGITUDE)]

# generate interactive leaflet map of elevators in NYC
p <- leaflet(coord_pairs) %>%
     addTiles('http://{s}.basemaps.cartocdn.com/dark_all/{z}/{x}/{y}.png', 
     attribution='Map tiles by <a href="http://stamen.com">Stamen Design</a>,
     <a href="http://creativecommons.org/licenses/by/3.0">CC BY 3.0</a> Map data
     <a href="http://www.openstreetmap.org/copyright">OpenStreetMap</a>') %>% 
     setView(-73.977896, 40.731724, zoom = 12) %>% 
     addCircles(~LONGITUDE, ~LATITUDE, weight = 3, radius=5, color="#ffa500", stroke = TRUE, fillOpacity = 0.8) %>% addLegend("bottomright", colors= "#ffa500", labels="Dunkin'", title="In Connecticut")

saveWidget(p, file='elevator_map_nyc.html')

The biggest liars in US politics

Sat, 11 Jun 2016 00:00:00 +0000

Who lies the most in US politics?

Most Americans, and anyone that follows US politics, will be aware of the tremendous changes and volatility that has struck the US political landscape in the past year. The ascent of Donald Trump from a billionaire entertainer to a fully fledged presidential candidate, alongside the unexpected popularity of Bernie Sanders and the nomination of Hillary Clinton as the first female presidential candidate, has catapulted this political season into one of the most entertaining, vicious and confusing year.

There have been a number of outlandish claims made recently, and one of the most important contribution of journalistic work is to comment and verify the veracity of statements made by those major political figures. One example of such work is published and maintained by Politifact, which regularly fact-checks statements made by a variety of sources, and displays the information in the following way:

I have casually browsed this website for a while, but I recently decided to try and surface some more information out of it. In particular, I wanted to:

visualize a summary of the overall distribution of statement veracity for each individual.
build a web app to visualize direct comparisons of statements between individuals.

In the following post, I will collect and analyze some of the Politifact data, while also going through the steps that allowed me to create the web application displayed below

Obtaining data on political statements

To begin, I wrote a Python script to scrape all the required data on Politifact. The code snippet below iterates through each page of the Politifact website to extract and process the HTML source code containing the following data points:

the individual that made the statement
a URL to a headshot of the individual
a categorical variable that states whether the statement is deemed to belong in one of the following categories (True; Mostly True; Half-True; Mostly False; False; Pants on Fire!; No Flip; Half Flip; Full Flop).

Below is the source code used to scrape the Politifact data

import urllib2
from bs4 import BeautifulSoup
import pandas as pd
import re
import json
import string
import itertools


def get_page_source(page):
    '''
    get HTML content of page
    Params
    args: 
    returns:
    '''
    try:
        url = 'http://www.politifact.com/truth-o-meter/statements/?page={}'.format(page)
        response = urllib2.urlopen(url)
        page_source = response.read()
        soup = BeautifulSoup(page_source, "html5lib")
        return soup
    except:
        print "Could not obtain data for page {}".format(page)
        return None


def extract_truth(table,
                  truth_meter,
                  image_source,
                  statement_text):
    '''extract statement text, image of candidate and truth of statement'''
    for truth in table:
        fact = [item['alt'] for item in truth.findAll("img")]
        images = [item['src'] for item in truth.findAll("img") if '.jpg' in item['src']]
        statement = [item.text.strip() for item in truth.findAll("a", {"class": "link"})]
        if fact[1] in truth_meter:
            truth_meter[fact[1]].append(fact[0])
            statement_text[fact[1]].append(statement[0])
        else:
            truth_meter[fact[1]] = []
            truth_meter[fact[1]].append(fact[0])
            statement_text[fact[1]] = []
            statement_text[fact[1]].append(statement[0])
            image_source[fact[1]] = images[0]
    return truth_meter, image_source, statement_text


def process_page_data(soup,
                      truth_meter,
                      image_source,
                      statement_text):
    try:
        table = soup.findAll("div", {'class': 'scoretable__item'})
        truth_meter, image_source, statement_text = extract_truth(table,
                                                                  truth_meter,
                                                                  image_source,
                                                                  statement_text)
        return truth_meter, image_source, statement_text
    except:
        return truth_meter, image_source, statement_text


def main():
    truth_meter, image_source, statement_text = {}, {}, {}
    for page in range(1, 198):
        print page
        soup = get_page_source(page)
        if soup is not None:
            truth_meter, image_source, statement_text = process_page_data(soup,
                                                                          truth_meter,
                                                                          image_source,
                                                                          statement_text)

    # process statement strings for R
    statement_text_processed = {}
    for k, val in statement_text.items():
        val_no_punctuation = ["".join(l for l in sent if l not in string.punctuation) \
                              for sent in val]
        val_lower = [sent.lower() for sent in val_no_punctuation]
        if len(val_lower) > 1:
            val_tokenized = ' '.join(val_lower).split(' ')
        else:
            val_tokenized = val_lower[0].split(' ')
        statement_text_processed[k] = val_tokenized

    with open('politifact_statements.txt', 'w') as outfile:
        json.dump(truth_meter, outfile)

    with open('politifact_statements_text.txt', 'w') as outfile:
        json.dump(statement_text_processed, outfile)

    df = pd.DataFrame.from_dict(image_source.items())
    df.to_csv('politifact_image_source.csv', index=False, encoding='utf-8')


if __name__ == '__main__':
    main()

A quick analysis of the Politifact data

With the data all gathered, I proceeded to perform a small analysis of the data. To begin, I looked at the different types of comments stated by the entities in the Politifact data. For example, what proportion of Donald Trump’s comments were True or False? Or what proportion of Donald Trump’s comments were a complete U-turn to his previous comments? For the sake of simplicity, I restricted the analysis to the 20 most common entities in the Politifact data, which produced the plot below:

So what do we see? As expected, “facts” reported through blogs or social media networks (Facebook posts) tend to be predominantly false. On the reverse, Democrats (Hillary Clinton, Barack Obama, Bernie Sanders) tend to speak truthful comments. It is interesting to note that establishment Republicans such as Jeb Bush, Marco Rubio, Scott Walker and Rick Scott also trend towards true comments, but have a non-negligible proportion of “Mostly False” comments. The conservative-leaning democrat Charlie Crist follows a similar pattern. Finally, the two most “anti-establishement” Republicans, Donald Trump and Ted Cruz, blow out their fellow politicians with a majority of “False” or “Mostly False” comments.

The figure below shows an aggregated view of the total proportion of True and False comments made by each entity, which clearly shows the disparities between Democrats and Republicans.

At this point, it is important to raise a potential caveat of this analysis, which is that we are unaware of amy potential bias within the Politifact data. It is conceivable that the people curating the comments are democratic-leaning, or that the sample of comments taken for each entity is not representative. My prior opinion makes me think that this is not the case, but unfortunately I have no data to back this up.

Building the Politifact web app

The next step was to build a Shiny app that would allow anyone to perform head-to-head comparisons between individuals in the Politifact data. For simplicity, I have hosted the app on shinyapps.io, but in order to ensure full reproducibility, I have also “containerized” the app using Docker. If you have not used or heard of Docker, their own website gives a very good description:

Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.`

With that in mind, the simplistic development workflow used to create the web app was:

Build the shiny app locally (i.e. on your local machine) and ensure that everything works just as expected.
Create a file called Dockerfile, which we will use to place all the relevant instructions and commands, as well as the environnment variables and configurations that will define our container.
Build and run the Docker container to check that the app behaves as expected. If not, iterate over the steps above until you are satisfied.
Reap the benefits of fully reproducible software across any computing platform!

The source code for the full web app can be found at the following GitHub repository, and has the following directory structure:

.
├── Dockerfile
├── README.md
├── app
│   ├── politifact_image_source.csv
│   ├── politifact_statements.txt
│   ├── politifact_statements_text.txt
│   ├── server.R
│   └── ui.R
├── requirements.txt
├── scrape_politifact.py
└── shiny-server.sh

1 directory, 10 files

By cloning this Github directory and typing the commands below, you will be able to have a fully working web app.

# build Docker container with tag name pol_app
docker build -t pol_app .

# run Docker container and mirror port 3838
docker run -p 3838:3838 -t pol_app

_{Note: the above was tested on a Ubuntu machine hosted on DigitlOcean, I’m assuming you will have to slightly tweak things when working on a local machine (i.e. if using something like Boot2Docker.}

A snapshot of the app is displayed below. It allows to select two political entities and highlights words most often stated by each, along with the overall distribution of types of comments (true, false, etc…) that they made.

Conclusions

We scraped and analyzed Politifact data, a resource that fact-checks statements made by various political entities. We found that statements conveyed via blogs or social media were often misleading. Interestingly, we also uncovered distinct differences between major political figureheads within the Republican and Democratic parties. While it is important to note that the inherent source data may be biased, we found that Democrats were generally more truthful in their political statements. To enable further deep dive of the data, we also build a light-weight app that enables head-to-head comparisons between any pair of entities in the Politifact data. In doing so, we obtained summarized views of the veracity of comments made by entities, as well as the broad topics that they cover.

Data science with Docker

Sat, 30 Apr 2016 00:00:00 +0000

Using docker to facilitate your data science pipelines

Until recently, and like many other fellow data scientists I have talked to, I built data science pipelines on my local machine or a remote host while relying on virtual environments. In doing so, I ensured some degree of replicability by keeping check of language versions, library versions, and so on. While this is a fairly efficient process that facilitated shared coding and portability, I do occasionally run into a common, recurring error.

For better or for worse, much of my development is performed on my local machine, and on the occasion where datasets or RAM requirements are too large, I will rely on a DigitalOcean droplet or EC2 instance to resolve that problem. However, neither of these cases tend to be the final step, as I will often refactor/modify my code in order to productionalize it into something that won’t terrify the code delivery team. In many cases, this will involve transitioning to a different machine or OS, such as moving from MAC OS to Ubuntu 14.04, and this is where the pain points with virtual environments appear. Often I will find myself configuring the OS to my requirements, installing the correct Python version, gcc compilers and BLAS libraries, which invariably leads to minor (sometimes major) conflicts, bugs and annoyances. More importantly, there is no way to really persist all these efforts should you wish to run this on another machine or have someone reproduce your code independently.

Docker solves all these issues!

What does Docker do?

Paraphrasing from their website,

Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

In other words, Docker offers the ability to contain ALL your code and dependencies into a single container, and never have to worry about it ever again (actually those were exactly the same words, but less well-written…). This was an intriguing concept, and the more qualified engineers around me assured me that it was also a good concept, so I decided to integrate this as part of my regular workflow. In the following, I will go through the steps to get started and create your first Docker container for a data science project, and hopefully convince you that Docker should be integrated as part of your regular workflow along the way!

If you haven’t already, you can install Docker on your local machine using the instructions at the following link https://docs.docker.com/engine/installation/mac/. Once that is done, you can go ahead and find the Docker Quickstart Terminal icon, double-click on it, and see a new terminal window appear with the following output:

Last login: Tue Feb 16 21:43:51 on ttys005
bash --login '/Applications/Docker/Docker Quickstart Terminal.app/Contents/Resources/Scripts/start.sh'
TLV-private:~ ThomasVincent$ bash --login '/Applications/Docker/Docker Quickstart Terminal.app/Contents/Resources/Scripts/start.sh'
Machine default already exists in VirtualBox.
Starting machine default...
(default) Starting VM...
Started machines may have new IP addresses. You may need to re-run the `docker-machine env` command.

Regenerate TLS machine certs?  Warning: this is irreversible. (y/n): Regenerating TLS certificates
Detecting the provisioner...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Setting environment variables for machine default...

                    ##         .
              ## ## ##        ==
           ## ## ## ## ##    ===
       /"""""""""""""""""\___/ ===
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
       \______ o           __/
         \    \         __/
          \____\_______/

docker is configured to use the default machine with IP XXX.XXX.XX.XXX
For help getting started, check out the docs at https://docs.docker.com

TLV-private:~ ThomasVincent$

Now that Docker is up and running, we can begin to build the main components of our container. For now, we will assume that you have a Python script that you want to automatically run every time you run a Docker container (although as we will see, other use cases can easily be adapted).

The basics

The very simplistic workflow for creating your own Docker container is the following:

Create a file called Dockerfile, which we will use to place the relevant instructions and commands that will define what our container can do.
Build the Docker container, this will install and apply all the environnment variables and configurations that you have specified in the Dockerfile.
Run the Dockerfile and reap the benefits of fully reproducible research across any computing platform!

Writing your own Dockerfile

A Dockerfile is usually started by specifying a base image from which you would like to start. Here, we will use the Ubuntu 14.04 image. It is also customary to specify the maintainer/creator of the Docker container you are about to create.

# specifiy base image
FROM ubuntu:14.04

# provide creator/maintainer of this Dockerfile
MAINTAINER Thomas Vincent <your_email_address@something.com>

The ubuntu:14.04 image comes as a very barebones implementation of the Linux OS, so it is up to us to specify some of the useful system tools and libraries that we would like to include:

# Update the sources list
RUN apt-get update

# install useful system tools and libraries
RUN apt-get install -y libfreetype6-dev && \
    apt-get install -y libglib2.0-0 \
                       libxext6 \
                       libsm6 \
                       libxrender1 \
                       libblas-dev \
                       liblapack-dev \
                       gfortran \
                       libfontconfig1 --fix-missing

RUN apt-get install tar \
                    git \
                    curl \
                    nano \
                    wget \
                    dialog \
                    net-tools \
                    build-essential

At this point, we are in a good position to also install the Python and the pip Python package manager.

# install Python and pip package manager
RUN apt-get install -y python \
                       python-dev \
                       python-distribute \
                       python-pip

As pip is installed, we can now go ahead and install commonly used Python libraries and other more specific ones that may be required by your Python script or web app.

# intall useful and/or required Python libraries to run your script
RUN pip install matplotlib \
                seaborn \
                pandas \
                numpy \
                scipy \
                sklearn \
                python-dateutil \
                gensim

Once you are sure that all the required Python libraries have been added, you can go ahead and define the command that you would like to when your Docker conatiner starts. In this case, we specify to run your Python script.

# define command to when Docker container starts
ENTRYPOINT ["python"]
CMD ["my_script.py"]

In all, your Dockerfile should look like this

# specifiy base image
FROM ubuntu:14.04

# provide creator/maintainer of this Dockerfile
MAINTAINER Thomas Vincent <your_email_address@something.com>

# Update the sources list
RUN apt-get update

# install useful system tools and libraries
RUN apt-get install -y libfreetype6-dev && \
    apt-get install -y libglib2.0-0 \
                       libxext6 \
                       libsm6 \
                       libxrender1 \
                       libblas-dev \
                       liblapack-dev \
                       gfortran \
                       libfontconfig1 --fix-missing

RUN apt-get install tar \
                    git \
                    curl \
                    nano \
                    wget \
                    dialog \
                    net-tools \
                    build-essential

# install Python and pip package manager
RUN apt-get install -y python \
                       python-dev \
                       python-distribute \
                       python-pip

# intall useful and/or required Python libraries to run your script
RUN pip install matplotlib \
                seaborn \
                pandas \
                numpy \
                scipy \
                sklearn \
                python-dateutil \
                gensim

# define command to when Docker container starts
ENTRYPOINT ["python"]
CMD ["my_script.py"]

Considering how much Docker can simplify your life, this is a pretty small and straighforward amount of code to have to write!

Building and running your Docker container

Now that you have written your own Dockerfile, you can go ahead and build the entire image. If you haven’t already, follow the instructions at the beginning of this post to ensure that you are running from within the Docker Terminal. Be sure that you are also working from within the directory in which your Dockerfile and Python script are located. You can then build your first Docker image by simply typing:

docker build -t your_image_name .

The -t flag allows you to name the docker container as whatever pleases you. When pressing enter, you should see a fair amount of logging information and other miscallaneous output displayed to your console. Once the build is finished, you can then go straight ahead and run your Docker image, which will compute or output whatever your Python script was intended to do!

docker run -t your_image_name

Shutting down your docker container

Once you are done with your docker container, you can shut it down by using the command

docker rm -f CONTAINER_ID

where the CONTAINER_ID value can be found by running

docker ps

Conclusions

We have covered a simple use case for Docker that allowed us to produce a running instance of a Python script that - along with all of its required dependencies - can be seamlessly ran across any system. In many ways, we have only scratched the surface of what can be done with Docker, but hopefully this demonstrates how useful it can be (and that is why it has seen an explosive growth in adoption across all segments of the tech industry!)

Player and roster similarity in the NBA

Thu, 17 Mar 2016 00:00:00 +0000

Recently, professional sports associations and teams have made big strides towards leveraging data to inform both personel and on-the-field decision making. While the four major leagues (NBA, NFL, MLB, NHL) vary in terms of where they are in that process, most people would argue that the NBA is at the forefront of this movement. If you have never heard of SportVu before, they are a company that has partnered with the NBA to “utilize a six-camera system installed in basketball arenas to track the real-time positions of players and the ball 25 times per second. Utilizing this tracking data, SportVu is able to create a wealth of innovative statistics based on speed, distance, player separation and ball possession.” As stated, the release of aggregated SportsVu data has offered brand new insights into how the game of basketball is played, and more importantly, how each individual plays the game.

In this post, I looked at the SportVu data available for all NBA players active during the 2014-2015 season. More particularly, I was interested in finding out whether SportVu data could be leveraged to discover players with similar playing styles, but also to discover teams with similar rosters. To begin, I started off by writing a quick Python script to scrape SportsVu data from http://www.stats.com/sportvu/sportvu-basketball-media/

Collections and scraping the data

# import required libraries for scraping and analysis

import urllib2
import json
import pandas as pd
from time import time
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import offsetbox
from sklearn.decomposition import TruncatedSVD
from sklearn import (manifold, datasets, decomposition, ensemble, lda,
                     random_projection)

# define endpoints from stats.nba.com that we wish to scrape

addressList = {
    "pullup_address": "http://stats.nba.com/js/data/sportvu/2014/pullUpShootData.json",
    "drives_address": "http://stats.nba.com/js/data/sportvu/2014/drivesData.json",
    "defense_address": "http://stats.nba.com/js/data/sportvu/2014/defenseData.json",
    "passing_address": "http://stats.nba.com/js/data/sportvu/2014/passingData.json",
    "touches_address": "http://stats.nba.com/js/data/sportvu/2014/touchesData.json",
    "speed_address": "http://stats.nba.com/js/data/sportvu/2014/speedData.json",
    "rebounding_address": "http://stats.nba.com/js/data/sportvu/2014/reboundingData.json",
    "catchshoot_address": "http://stats.nba.com/js/data/sportvu/2014/catchShootData.json",
    "shooting_address": "http://stats.nba.com/js/data/sportvu/2014/shootingData.json"
    }

# download and format SportVu data, data for each endpoint is placed in a dictionnary

sportsvu_data = {}
for key, val in addressList.items():
    print key
    url = val
    response = urllib2.urlopen(url)
    raw = response.read()
    data = json.loads(raw)
    headers = data['resultSets'][0]['headers']
    table = data['resultSets'][0]['rowSet']
    df = pd.DataFrame(table)
    df.columns = headers
    df = df.set_index(['PLAYER_ID', 'TEAM_ABBREVIATION'])
    df.columns = ['{}_{}'.format(key, x) for x in df.columns]
    sportsvu_data[key] = df

# concatenate all sportsvu data

sportvu = None
for suffix, input_df in sportsvu_data.items():
    print suffix
    if sportvu is None:
        sportvu = input_df
    else:
        sportvu = pd.merge(sportvu,
                           input_df,
                           how='inner',
                           left_index=True,
                           right_index=True)

# filter out dataframe to players who have had relevant playing time

df = sportvu
df = df[df.pullup_address_GP >= 41]
df = df[df.pullup_address_MIN >= 15]

# set index of final dataframe

df_final = df.T.drop_duplicates().T.reset_index()
df_final = df_final.set_index(['PLAYER_ID', 'TEAM_ABBREVIATION', 'pullup_address_PLAYER'])

# remove those columns because they are redundant

cols_to_remove = ['pullup_address_FIRST_NAME', 'pullup_address_LAST_NAME',
                  'pullup_address_GP', 'pullup_address_MIN']
df_final.drop(cols_to_remove, axis=1, inplace=True)

# remove duplicate columns

df_final = df.T.drop_duplicates().T

There is nothing extraordinary in the code above, I essentially scraped some data from publicly (I hope) available data, and after a small clean-up, concatenated all the data into a single Pandas dataframe. The only thing of note is that I restricted the data collection to players who averaged at least 15 minutes per game and played in at least half the games in the season. In total, this leaves us with 329 NBA players, each with 80 unique Sportvu data points.

Inferring and visualizing similarities between NBA players

With the data now in our hands (or RAM), we can proceed to the original intent of this blog post, which is finding players with the most similar playing styles. After some trial and error, I obtained the best results when computing the correlation matrix between Sportvu metrics for all players, and then applying the t-SNE dimensionality reduction algorithm. Roughly, t-SNE is considered to be useful because of its property to conserve the overall topology of the data, so that neighboring (i.e. similar) players are mapped to neighboring locations in a two-dimensional space (It is this property that makes it so amenable to image analysis). Other well-known clustering techniques such as k-means or MDS would also be adequate for this exercise, but I’ve had good fortune when using t-SNE, so am perhaps unwisely sticking to it here.

# compute the correlation matrix between SportsVu metrics for all NBA players
# note: Pearson or Spearman showed little differences in results

df_final = df_final.convert_objects(convert_numeric=True)
df_final = df_final.astype(float)
corr = df_final.T.corr()
X = corr.as_matrix()

# perform t-SNE dimensionality reduction and print to file

X_reduced = TruncatedSVD(n_components=10, random_state=0).fit_transform(X)
tsne = manifold.TSNE(n_components=2, perplexity=40, verbose=2)
X_tsne = tsne.fit_transform(X_reduced)

df = pd.DataFrame(X_tsne)
df['player'] = [x[2] for x in df_final.index]
df['team'] = [x[1] for x in df_final.index]
df.columns = ['x', 'y', 'player', 'team']
df.to_csv('tsne_clusters.csv', index=False)

The advantage of using t-SNE in this context is that we are effectively taking an unsupervised approach, with the hopes that we can infer natural groupings of players based on their Sportvu statistics. Now that the data has been processed, we can start to visualize it. From there on, I will proceed to some nasty context switching and use R (I love both R and Python but hate using both in a single projetc. However, I am justifying my decision on the fact that, despite recent progress for Python, R still currently has far better wrappers around JS/D3).

# import required R libraries

library(data.table)
library(scatterD3)
library(d3heatmap)
library(ggplot2)
library(ggrepel)
library(htmlwidgets)

# read in tsne representations

dt <- fread('tsne_clusters.csv', header=TRUE)
dt <- dt[team!='TOTAL']

# Use k-means to uncover natural clusters of players
# Compute sum of squares for different values of k

wss <- c()
for (i in 1:15) {
wss <- c(wss, sum(kmeans(dt[, list(x, y)], centers=i)$withinss))
}

# plot sum of squares as a function of cluster count in order
# to find the "elbow". Optimal cluster count was found to be 7

plot(1:15, wss, type="b",
     xlab="Number of Clusters",
     ylab="Within groups sum of squares")

# add cluster assigments to data.table object

cl <- kmeans(dt[, list(x, y)], centers=7)
positions <- c('Shooting Guard', 'Dynamic C/PF',
               'Slow C/PF', 'Dynamic SF/PF',
               'Slow SF/PF', 'Point Guard', 'Stretch C/PF')
dt[, 'cluster' := cl$cluster]
dt[, 'position' := positions[dt$cluster]]

# plot and save interactive D3 scatter plot

tooltips <- paste("<strong>", dt$player,"</strong><br /><strong>", dt$team, "</strong><br />")
p <- scatterD3(x = dt$x,
          y = dt$y,
          lab = dt$player,
          col_var=dt$team,
          symbol_var=dt$position,
          point_opacity = 0.7,
          tooltip_text = tooltips,
          col_lab = "Team",
          symbol_lab = "Position Group",
          width=1000,
          height=1000)

saveWidget(p, file='nba_player_similarity.html')

The plot below shows the natural groupings of players, where the shape represents the cluster they belong to and the color represent their respective teams. Feel free to zoom, highlight certain teams or clusters (by hovering over the legends) and generally just playing around it.

NBA player similarity

Upon investigation, we can see that this approach makes a lot of sense. For example, players such as Damian Lillard, Mario Chalmers, Eric Bledsoe or Derick Rose are very to each other in space. There are many other examples like this (Serge Ibaka and Lamarcus Aldridge; Jimmy Butler and Andre Iguodola) but it is interesting to note how the shooting and point guard have well-defined positions, whereas the Center, Power Forward and Small Forward positions show a lot more heterogeneity and complexity. There are some mis-assignments here and there but these tend to be on the boundary of clusters, which could be probably be fixed after some further optimization and tinkering of the cluster assignments.

Team roster similarity

We can also leverage the results obtained from the dimensionality reduction part to discover teams that share the most similar roster of players. Given two teams X and Y with players [x1,..., xn] and [y1,...,yn] respectively, one way of achieving this is through the following steps:

Select player from team X (say x1)
Compute point-to-point distance between player x1 and all players [y1,…,yn] in team Y
Select and record the minimum value between player x1 and all players [y1,…,yn] in team Y. We are effectively finding the player in team Y that is most similar to player x1
Repeat step 1 to 3 for all remaining players in team X and sum the total point distance to get a “distance” value between team X and Y

By summing distances between pairs of players that are most similar in each teams, we can then assume that pairs of teams with low total distance between one another have more similar rosters than pairs of teams with high total distance.

# get point-point distance between teams

'point_distance' <- function(p1, p2) {
    dx = (p1[1] - p2[1])^2
    dy = (p1[2] - p2[2])^2
    return(sqrt(dx+dy))
}

unique.teams <- sort(unique(dt$team))
roster_distance <- mat.or.vec(length(unique.teams), length(unique.teams))
row.names(roster_distance) <- unique.teams
colnames(roster_distance) <- unique.teams
for(i in 1:length(unique.teams)) {
  print(unique.teams[i])
  for(j in 1:length(unique.teams)) {
    t1 <- dt[dt$team == unique.teams[i]]
    t2 <- dt[dt$team == unique.teams[j]]
    sim <- apply(t1, 1, function(x) 
                            apply(t2, 1,
                              function(y) point_distance(as.numeric(x[1:2]), as.numeric(y[1:2]))))
    roster_distance[i, j] <- mean(apply(sim, 2, min))
  }
}

p <- d3heatmap(roster_distance, colors = "Spectral", dendrogram="none")
saveWidget(p, file='nba_team_similarity.html')

NBA team roster similarity

Tracking Social Issues and Topics in Presidential Speeches

Fri, 23 Oct 2015 00:00:00 +0000

Scraping presidential transcripts

To begin, we must scrape the content of all presidential speeches recorded in American history. To do that, I’ll rely on the very handy BeautifulSoup library, and eventually store all data in a pandas dataframe that will be persisted in a pickle file.

 
# import required libraries to scrape presidential transcripts
from bs4 import BeautifulSoup
import pandas as pd
import pickle
import urllib2
import re

 
def get_speech_links():
    '''
    scrape content of pages with all presidential transcript links
    '''
    home_url = 'http://millercenter.org/president/speeches'
    try:
        response = urllib2.urlopen(home_url)
        page_source = response.read()
        soup = BeautifulSoup(page_source, "html5lib")
        transcript_links = soup.findAll("a", {'class': 'transcript'})
        return transcript_links
    except urllib2.HTTPError:
        print 'Homepage not available!'
        return None

 
def get_transcript(speech_link):
    '''
    scrape title of speech, date of speech and full transcipt
    contained in the input speech_link URL
    '''
    speaking = speech_link.split('/')[2]
    new_link = base_url + str(speech_link)
    try:
        response = urllib2.urlopen(new_link)
        page_source = response.read()
        soup = BeautifulSoup(page_source, "html5lib")
        title = soup.find('title').text
        speech_date = title.split('(', 1)[1].split(')')[0]
        transcript = soup.find('div', {'id': 'transcript'}).text
        transcript = transcript.replace('\n', ' ').replace('\r', '').replace('\t', '')
        return {'speaker': speaking,
                'date': speech_date,
                'title': title,
                'transcript': transcript}
    except urllib2.HTTPError:
        print 'skipped ' + str(speech_link)
        return None

# iterate through all links and extract content
transcript_links = get_speech_links()
base_url = 'http://millercenter.org/'
transcript_dict = {}
for i, link in enumerate(transcript_links):
    if i % 100 == 0:
        print 'Scraped ' + str(i) + '/' + str(len(transcript_links)) + ' of links...'
    if link.has_attr('href'):
        transcript_data = get_transcript(link['href'])
        if transcript_data is not None:
            key = transcript_data['speaker'] + '|' + transcript_data['date']
            transcript_dict[key] = transcript_data

 
# dump dataframe to pickle object
df = pd.DataFrame.from_dict(transcript_dict, orient='index')
pickle.dump(df, open( "presidential_speeches.pickle", "wb" ))

Topic modeling and visualization

Now that the raw text of all presidential speeches in American history has been retrieved, we can proceed to light preprocessing before applying Latent Dirichlet Allocation.

 
%matplotlib inline  
from gensim.parsing.preprocessing import STOPWORDS
from gensim.utils import simple_preprocess
from gensim.models import TfidfModel, LsiModel
from gensim.models.ldamodel import LdaModel
from gensim import corpora
from gensim import matutils
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans
from collections import defaultdict
import pyLDAvis.gensim as gensimvis
import pyLDAvis
import pandas as pd
import numpy as np
import pickle
import lda

In the following 5 cells, we effectively tokenize and remove stopwords from each document (i.e. presidential speech), compute the frequency of each token, and filter out all those that appear less than 10 times in the entire corpus of presidential speeches. Note that I used an ad-hoc threshold of 10, but this should be a parameter that could be played around. Also, the amount of porcessing on each document is intentionally simplistic. Finally, we set up gensim-specific objects that include a dictionary mapping words to integer ids, and a corpus that simply counts the number of occurences of each distinct word, converts the word to its integer word id and returns the result as a sparse vector.

 
def tokenize(text):
    return [token for token in simple_preprocess(text) if token not in STOPWORDS]

 
df = pickle.load( open( "presidential_speeches.pickle", "rb" ) )
speeches = df.transcript
documents = speeches.tolist()
texts = [tokenize(document) for document in documents]

 
frequency = defaultdict(int)
for text in texts:
    for token in text:
        frequency[token] += 1

 
texts = [[token for token in text if frequency[token] > 10] for text in texts]

 
dictionary = corpora.Dictionary(texts)
corpus = [dictionary.doc2bow(text) for text in texts]

Finding the Optimum Number of Topics

Now that the data is ready, we can run a batch LDA (because of the small size of the dataset that we are working with) to discover the main topics in our document.

# fit LDA model
speeches_topics = LdaModel(corpus=corpus,
                           id2word=dictionary,
                           num_topics=20,
                           passes=10)

# print out first 10 topics
for i, topic in enumerate(speeches_topics.print_topics(10)):
    print '{} --- {}'.format(i, topic)

0 --- 0.025*states + 0.017*united + 0.017*shall + 0.014*state + 0.010*constitution + 0.009*president + 0.009*act + 0.009*congress + 0.008*laws + 0.007*law

1 — 0.026government + 0.007states + 0.007chilean + 0.007men + 0.006sailors + 0.006united + 0.005mr + 0.005german + 0.005police + 0.004vessels

2 — 0.012world + 0.011peace + 0.009people + 0.008america + 0.008freedom + 0.007soviet + 0.006united + 0.006new + 0.005states + 0.005nations

3 — 0.050president + 0.030mr + 0.024think + 0.008secretary + 0.008general + 0.008people + 0.007time + 0.007viet + 0.007going + 0.007nam

4 — 0.011government + 0.007people + 0.007business + 0.006country + 0.005economic + 0.005congress + 0.005world + 0.005federal + 0.005tax + 0.005public

5 — 0.006people + 0.004government + 0.004united + 0.004states + 0.003country + 0.003public + 0.003congress + 0.003question + 0.003going + 0.002time

6 — 0.013states + 0.012government + 0.009united + 0.008congress + 0.007public + 0.005country + 0.005great + 0.005year + 0.004general + 0.004people

7 — 0.011peace + 0.010vietnam + 0.010people + 0.009war + 0.009world + 0.008united + 0.007south + 0.007american + 0.007nations + 0.006states

8 — 0.009world + 0.009congress + 0.008new + 0.008year + 0.007america + 0.006people + 0.006energy + 0.006american + 0.006nation + 0.005government

9 — 0.014government + 0.012people + 0.007states + 0.007union + 0.007constitution + 0.006great + 0.006shall + 0.006men + 0.006country + 0.005free

The display of inferred topics shown above does not really lend itself very well to interpretation. Aside from the fact that you have to read through all the topics, most people will interpret the main themes of each topics differently. This hits right to the core of my mixed feelings towards topic modeling. To be given the ability and opportunity to infer topics from a large set of documents is truly amazing, but I have always personally felt (and maybe that is just me) that the ensuing display of information was lacking. Indeed, I have found that the output of typical topic modeling techniques does not lend itself very well to visualization and - in the case of presentations to the uninitiated - interpretation. However, I recently came across the LDAviz R library developed by Kenny Shirley and Carson Sievert, which to paraphrase their words is a D3.js interactive visualization that's designed help you interpret the topics in a topic model fit to a corpus of text using LDA. Here, we use the great Python extension port of the LDAviz R library, available on GitHub at the following URL https://github.com/bmabey/pyLDAvis. Two attractive features of pyLDAviz are its ability to help interpret the topics extracted from a fitted LDA model, but also the fact that it can be easily incorporated within an iPython notebook in nothing more than two lines of code!

 
vis_data = gensimvis.prepare(speeches_topics, corpus, dictionary)
pyLDAvis.display(vis_data)

Tracking and visualizing topics propensity over time

Now that we have shown how results gathered from topic modeling methods such as LDA can be visualized in a intuitive way, we can move to additional data analysis. In particular, it would be interesting to uncover the temporal variation of topics across American History. I would personally be very curious to find out whether topic modeling can reverse-engineer the major events in American History. In the next step, we produce a dataframe where each row represents a speech and each of the 20 columns represent a topic. Each cell in the dataframe represents the probability that a given topic was assigned to a speech.

 
# example command to print out main topics inferred in a presidential speech
print speeches_topics.get_document_topics(corpus[10])

[(7, 0.10493554997876908), (10, 0.011621459517891617), (15, 0.86674743636700446)]

 
# extract all document-topic distritbutions to dictionnary
document_key = list(speeches.index)
document_topic = {}
for doc_id in range(len(corpus)):
    docbok = corpus[doc_id]
    doc_topics = speeches_topics.get_document_topics(docbok, 0)
    tmp = []
    for topic_id, topic_prob in doc_topics:
        tmp.append(topic_prob)
    document_topic[document_key[doc_id]] = tmp

 
# convert dictionnary of document-topic distritbutions to dataframe
df = pd.DataFrame.from_dict(document_topic, orient='index')

 
president_speech = [x.split('|')[0] for x in df.index]
year_speech = [y.split(',')[1] if ',' in y else None for y in [x.split('|')[1] for x in df.index]]

 
topics_speech = df
topics_speech['president'] = pd.Series(president_speech, index=df.index)
topics_speech['year'] = pd.Series(year_speech, index=df.index)
topic_column_names = ['topic_' + str(i) for i in range(0, 20)]
topic_column_names.append('president')
topic_column_names.append('year')
topics_speech.columns = topic_column_names
print topics_speech.head(5)
# topics_speech.to_csv('topics_by_speech.csv', sep=',')

                              topic_0   topic_1   topic_2   topic_3   topic_4  \
lincoln|July 4, 1861         0.000011  0.000011  0.000011  0.011112  0.000011   
buchanan|February 24, 1859   0.000033  0.000033  0.000033  0.203170  0.000033   
reagan|November 11, 1988     0.000526  0.197846  0.000526  0.000526  0.000526   
tyler|February 20, 1845      0.000114  0.000114  0.000114  0.000114  0.000114   
eisenhower|January 17, 1961  0.000063  0.000063  0.000063  0.000063  0.000063   

                              topic_5   topic_6   topic_7   topic_8   topic_9  \
lincoln|July 4, 1861         0.008504  0.000011  0.064621  0.051711  0.752269   
buchanan|February 24, 1859   0.000033  0.000033  0.002136  0.249423  0.032370   
reagan|November 11, 1988     0.453219  0.000526  0.000526  0.000526  0.000526   
tyler|February 20, 1845      0.000114  0.000114  0.014633  0.361549  0.128588   
eisenhower|January 17, 1961  0.615735  0.000063  0.000063  0.000063  0.000063   

                             ...    topic_12  topic_13  topic_14  topic_15  \
lincoln|July 4, 1861         ...    0.000011  0.000011  0.000011  0.111633   
buchanan|February 24, 1859   ...    0.000033  0.000033  0.006279  0.490955   
reagan|November 11, 1988     ...    0.101381  0.239133  0.000526  0.000526   
tyler|February 20, 1845      ...    0.000114  0.000114  0.000114  0.493404   
eisenhower|January 17, 1961  ...    0.000063  0.074416  0.000063  0.000063   

                             topic_16  topic_17  topic_18  topic_19  \
lincoln|July 4, 1861         0.000011  0.000011  0.000011  0.000011   
buchanan|February 24, 1859   0.015235  0.000033  0.000033  0.000033   
reagan|November 11, 1988     0.000526  0.000526  0.000526  0.000526   
tyler|February 20, 1845      0.000114  0.000114  0.000114  0.000114   
eisenhower|January 17, 1961  0.010800  0.000063  0.000063  0.000063   

                              president   year  
lincoln|July 4, 1861            lincoln   1861  
buchanan|February 24, 1859     buchanan   1859  
reagan|November 11, 1988         reagan   1988  
tyler|February 20, 1845           tyler   1845  
eisenhower|January 17, 1961  eisenhower   1961  

[5 rows x 22 columns]

Finally, we can compute the normalized frequency of topics by year and plot these as a time-series using the dygraphs library.

 
columns = ['topic_'+str(i) for i in range(0, 20)] # define columns to process
df = pd.DataFrame(topics_speech.groupby('year')[columns].sum()) # group topics frequency by year
df = 100 * df.div(df.sum(axis=1), axis=0) # normalize topic frequencies by year
df = np.round(df, 1) # round topic frequencies
df.to_csv('topics_by_year.csv', sep=',')

At this point, I’m going to do something that I am not very proud of and proceed to some nasty context switching. Although I played around with the charts library, I was not satisified with the results and temporarily switched to R in order to leverage the dygraphs library. Thankfully, Jupyter notebooks have plenty of magic that make it easy to call R from the notebook itself!

 
import rpy2

 
%load_ext rpy2.ipython

 
%%R
library(htmlwidgets)
library(dygraphs)
library(zoo)
library(xts)

dt <- read.zoo("topics_by_year.csv", sep = ",", index.column = 1, header = TRUE, format = "%Y")
dt_large <- dt[, which(apply(dt, 2, sum)>=500)]
dt_large$others <- 100- apply(dt_large, 1, sum)
dt_large <- as.xts(dt_large)

 
%%R
dygraph(dt_large) %>%
dyAnnotation("1970-1-1", text = "Vietnam") %>%
  dyShading(from = "1961-1-1", to = "1975-1-1", color = "#bdbdbd")  %>%
  dyShading(from = "1938-1-1", to = "1945-1-1", color = "#bdbdbd")  %>%
  dyShading(from = "1914-1-1", to = "1919-1-1", color = "#bdbdbd")  %>%
  dyShading(from = "1861-1-1", to = "1865-1-1", color = "#bdbdbd")  %>%
  dyRoller(rollPeriod = 8) %>%
  dyOptions(stackedGraph = TRUE) %>%
  dyRangeSelector(height = 20) %>%
  dyLegend(show = "follow") %>%
  #dyLegend(width = 900) %>%
  dySeries("topic_1", label = "(sailors/chili)") %>%
  dySeries("topic_4", label = "(world/countries)") %>%
  dySeries("topic_6", label = "(law/congress)") %>%
  dySeries("topic_7", label = "(bank/silver/bank)") %>%
  dySeries("topic_10", label = "(work/people/children)") %>%
  dySeries("topic_12", label = "(peace/war)") %>%
  dySeries("topic_13", label = "(law/constition/congress)") %>%
  dySeries("topic_15", label = "(public)") %>%
  dySeries("topic_18", label = "(business/economic/tax)") %>%
  dySeries("others", label = "(soviet/german/war)")

Tracking and visualizing topics propensity over time

Clustering individual presidential speeches

We can also wrangle the data a little bit more in order to visualize how each individual speeches cluster together. This time, we use document-topic distributions and apply the t-sne dimensionality reduction algorithm to map all speeches into two-dimensional space. Roughly, t-sne is considered to be useful because of its property to conserve the overall topology of the data, so that neighboring (i.e. similar) speeches will hopefully be mapped into neighboring locations in two-dimensional space. Other well-known clustering techniques such as k-means or MDS would likely be just as adequate for this exercise, but I’ve had good fortune when using t-sne, so am unwisely (and perharps not very smartly) sticking to it here.

 
# perform T-SNE on dataframe of document-topic distritbutions
# the aim is to show a 2-D representation of presidential speeches based on their inferred topics
from sklearn import manifold
from time import time
t0 = time()
tsne = manifold.TSNE(n_components=2, init='pca', random_state=0)
Y = tsne.fit_transform(df)
t1 = time()
print("t-SNE: %.2g sec" % (t1 - t0))

t-SNE: 13 sec

 
# output T-SNE 2-D representation of each speech to file
president_clusters = pd.DataFrame(Y)
president_clusters = president_clusters.set_index(df.index)
print president_clusters.head(5)
president_clusters.to_csv('speech_tsne_clusters.csv', sep=',')

                                     0         1
lincoln|July 4, 1861         12.246211 -4.594903
buchanan|February 24, 1859   13.982249 -1.675186
reagan|November 11, 1988     -7.665759  4.714818
tyler|February 20, 1845      11.953091  1.884652
eisenhower|January 17, 1961 -13.193183 -3.790267

We can now leverage the mpld3 library to display the t-sne clusters inline. The interactive figure below shows the 2-dimensional t-sne coordinates of all 880 presidential speeches in American history. One of the challenges here was to generate distinct colors to map the different presidents, and I don’t think I did a particularly good job at it (the figure could probably benefit from a legend too, but I opted to waste my time on adding tooltip functionnality instead!)

import matplotlib.pyplot as plt
import numpy as np
import mpld3
import colorsys
from mpld3 import plugins
mpld3.enable_notebook()

presidents = [x.split('|')[0] for x in president_clusters.ix[:,0]]
unique_presidents = list(set(presidents))
N = len(unique_presidents)
HSV_tuples = [(x*1.0/N, 0.7, 0.9) for x in range(N)]
RGB_tuples = map(lambda x: colorsys.hsv_to_rgb(*x), HSV_tuples)

 
president_colors = {}
for i, p in enumerate(unique_presidents):
    president_colors[p] = RGB_tuples[i]

 
fig, ax = plt.subplots(subplot_kw=dict(axisbg='#EEEEEE'), figsize=(10,10))
ax.grid(color='white', linestyle='solid')
scatter = ax.scatter(president_clusters.ix[:, 1],
                  president_clusters.ix[:, 2],
                     s=80,
                     label=presidents,
                     alpha=0.3,
                  c=[president_colors[p] for p in presidents])

ax.grid(color='white', linestyle='solid')
ax.set_title("Clustering presidential speeches", size=20)
labels = ['<h3>{president}</h3>'.format(president=x) for x in presidents] 
tooltip = plugins.PointHTMLTooltip(scatter, labels)
plugins.connect(fig, tooltip)

Clustering presidents

Finally, we can look at how each president cluster with one another based on the entire corpus of their speeches. Since we are clustering based on their speeches, it would be reasonable to expect that presidents with similar ideologies, or that were confronted to similar historical situations such as internal or external conflicts, economic depression or periods of significant political instability, should cluster close to one another. For this exercise, we use the CountVectorizer function from the scikit-learn library to generate president-specific corpuses built using all their speeches. As a result, we obtain a term-frequency matrix where each row is a president, columns are words, and cells represents the total number of times a president used a given word. Note that I mostly relied on defaut parameters for the CountVectorizer, but we could potentially change a lot of parameters including stopwords list, minimum/maximum document-frequency, maximum number of words, n-grams generations and more…

df = pickle.load( open( "presidential_speeches.pickle", "rb" ) )
speeches = df.transcript.tolist()
presidents = df.speaker.tolist()

from collections import defaultdict
president_words = defaultdict(list)
for i, president in enumerate(presidents):
    president_words[president].append(speeches[i])

president_all_words = []
for key in president_words.keys():
    president_all_words.append(' '.join(president_words[key]).encode('utf8'))

from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer(min_df=2,
                             ngram_range=(1, 2),
                             strip_accents='unicode',
                             stop_words='english',
                             max_features=5000)
X = vectorizer.fit_transform(president_all_words)
vocab = vectorizer.get_feature_names()
print X.shape

# output T-SNE 2-D representation of each speech to file
president_clusters = pd.DataFrame(Y)
president_clusters['president'] = pd.Series([str(x) for x in president_words.keys()], index=president_clusters.index)
print president_clusters.head(5)
president_clusters.to_csv('president_tsne_clusters.csv', index=False)

             0           1 president
-504.216414 -139.964997   madison
 129.954414  639.936735      taft
1452.238353 -825.087538   clinton
 735.346376 -492.296639    carter
-147.329302  433.361482  buchanan

# extract unique lits of presidents
presidents = [x for x in president_clusters['president']]
unique_presidents = list(set(presidents))
N = len(unique_presidents)
HSV_tuples = [(x*1.0/N, 0.5, 0.9) for x in range(N)]
RGB_tuples = map(lambda x: colorsys.hsv_to_rgb(*x), HSV_tuples)

# map distinct color to each president
president_colors = {}
for i, p in enumerate(unique_presidents):
    president_colors[p] = RGB_tuples[i]

fig, ax = plt.subplots(subplot_kw=dict(axisbg='#EEEEEE'), figsize=(10,10))
ax.grid(color='white', linestyle='solid')
scatter = ax.scatter(president_clusters.ix[:, 0],
                  president_clusters.ix[:, 1],
                     s=80,
                     label=presidents,
                  c=[president_colors[p] for p in presidents])

ax.grid(color='white', linestyle='solid')
ax.set_title("Clustering presidents based on their speeches", size=20)
labels = ['<h3>{president}</h3>'.format(president=x) for x in presidents] 
tooltip = plugins.PointHTMLTooltip(scatter, labels)
plugins.connect(fig, tooltip)

Cloning a graph in Python

Sat, 10 Oct 2015 00:00:00 +0000

If you have ever played around with Algorithms & Data Structures, then you most likely have heard of Leetcode.com, which contains a number of famous (or infamous) of technical questions. One of my favorite in there is the graph clone question, which can be shortly stated as:

Clone an undirected graph. Each node in the graph contains a label and a list of its neighbors.

Like most algorithms & data structures problems, there are usually many variants to solving this problem, but here I will go through the solution that is most typically reported. To begin, it is generally good form to define a Node class:

1
2
3
4
5
6
7
8
 
class Node:
    '''
    Define a node class for our undirected graph
    '''
    def __init__(self, node_label, node_neighbors=[]):
        self.label = node_label
        self.neighbors = node_neighbors

You can test the Node class in your Python interpreter as below (although this would never be considered a test per-se if we are talking in engineering terms!)

1
2
3
4
5
6
 
>>> test_node = Node('test')
>>> print test_node.label
test
>>> print test_node.neighbors
[]

Finally, we can define the Solution class, which will allow us to clone any graph given that we are provided at least one of its node:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
 
class Solution:
    '''
    Clone undirected graph
    '''
    def cloneGraph(self, node):
        nodeMap = {}
        return self.cloneNode(node, nodeMap)
    
    def cloneNode(self, node, nodeMap):
        if node == None:
            return None
        if node.label in nodeMap:
            return nodeMap[node.label]
        newNode = Node(node.label)
        nodeMap[node.label] = newNode
        for neighbor in node.neighbors:
            newNode.neighbors.append(self.cloneNode(neighbor, nodeMap))
        return newNode

So what does the code above do? Quite clearly, the cloneGraph function in lines 5-7 takes as input a node belonging to the graph we wish to clone, and then initializes a hash map (or dictionnary if we are talking in Python terms). Finally it invokes the cloneNode function that is defined in lines 9-18. First, the cloneNode function checks for an empty node and if this statement is met returns a None value. The next step (lines 12-13) is to check whether the input node is already in nodeMap, which would indicate it has already been processed. If this condition is met, then we simply return label of the node (for reasons that will become clear in the next paragraph).

At this point, we get to the meaty part of our solution. Line 14 with the statement newNode = Node(node.label) initiates a new node with the same label as the input node and no neighbors (the default for the Node class). Next, we add the newly created newNode object to the hash map nodeMap. Finally, we iterate through all known neighbors of the input node, and recursively append the results of calling cloneNode to each of those neighbors. In doing so, nodes already cloned in nodeMap will simply be appended to the neighbor list of newNode (because of the conditional statement in lines 12-13), while all others will also be recursively added to the nodeMap hash map. This will have the direct effect of cloning the entire original graph to the nodeMap hashmap! And we are done!