Skip to main content

Lollapalooza according to Spotify

analysis
spotify
lollapalooza
api
Web scraping the Lollapalooza São Paulo 2017 schedule and joining it with band popularity from the Spotify API, in R.
Author

Athos Damiani

Published

March 27, 2017

On the day I went to Lollapalooza I discovered Rspotify, a wrapper for the Spotify API, and that gave me the idea of putting together info from both.

The fun here will involve

This year’s Lollapalooza in São Paulo had 47 bands spread over four stages. The fun part is linking the Lolla schedule to the band popularity info provided by Spotify. Below I describe how I got the data, list the three hypotheses I came up with and make some plots to discuss them.

Dataset

Prerequisites

Packages

# installs Rspotify
if(!require("Rspotify"))
  devtools::install_github("tiagomendesdantas/Rspotify")
library(Rspotify)
library(magrittr)
library(forcats)
library(stringi)
library(lubridate)
library(httr)
library(rvest)
library(tidyverse)

Rspotify is a new package that isn’t on CRAN yet, but it already works.

Spotify account

To use the Spotify API you need an account on their website, as you can imagine.

Spotify App

To get an access code for their API you need to create an App within your Spotify account; this is the most bureaucratic prerequisite of all. I learned how to do it by following the steps in the README of the Rspotify package itself on Github (see here).

In the end, you’ll have an app_id, a client_id and a client_secret in hand.

Extracting the Lollapalooza SP 2017 schedule

The goal here is to get a data.frame version of the tables on the website lollapaloozabr.com/lineup-horarios/. It has the full schedule for both days of the event.

On to the code! Tip: the best way to learn what each step of the code does is to run it line by line and look at the result.

# schedule from the lollapalooza 2017 website ------------------

lolla2017_programacao <- "https://www.lollapaloozabr.com/lineup-horarios/" %>%
  GET() %>%
  read_html() %>%
  html_table() %>%
  set_names(c("sabado", "domingo")) %>%
  at_depth(2, ~ .x %>%
             stri_replace_all_regex(" {2,}", "") %>%
             stri_replace_all_regex("[\\n]{1}", ",") %>%
             stri_replace_all_regex("[,]{2,}", "\\\n") %>%
             read.csv(text = ., header = FALSE,
                      col.names = c("artist", "hora"))) %>%
  map(~ .x[-1] %>%
        data_frame(palco = names(.), programacao_palco = .) %>%
        unnest(programacao_palco)) %>%
  data_frame(dia = names(.), programacao = .) %>%
  unnest(programacao) %>%
  separate(hora, c("hora_ini", "hora_fim"), sep = "-") %>%
  mutate(artist = artist %>% tolower,
         hora_ini = paste(if_else(dia %in% "sabado", "2017-03-25", "2017-03-26"), hora_ini) %>% ymd_hm(),
         hora_ini = if_else(hour(hora_ini) < 12, hora_ini + hours(12), hora_ini),
         hora_fim = paste(if_else(dia %in% "sabado", "2017-03-25", "2017-03-26"), hora_fim) %>% ymd_hm(),
         hora_fim = if_else(hour(hora_fim) < 12, hora_fim + hours(12), hora_fim),
         dia = fct_relevel(dia, c("sabado", "domingo")))

Result

dia palco artist hora_ini hora_fim
sabado Palco Skol cage the elephant 2017-03-25 16:25:00 2017-03-25 17:25:00
sabado Palco Skol metallica 2017-03-25 21:00:00 2017-03-25 23:00:00
sabado Palco Skol doctor pheabes 2017-03-25 12:05:00 2017-03-25 13:05:00
sabado Palco Skol rancid 2017-03-25 18:35:00 2017-03-25 19:35:00
sabado Palco Skol suricato 2017-03-25 14:15:00 2017-03-25 15:15:00
sabado Palco Onix the 1975 2017-03-25 17:30:00 2017-03-25 18:30:00
sabado Palco Onix the outs 2017-03-25 13:10:00 2017-03-25 14:10:00
sabado Palco Onix the xx 2017-03-25 19:40:00 2017-03-25 20:55:00

It’s interesting to note that to build this simple little table we used the httr, rvest, purrr, dplyr, tidyr, lubridate, stringi and forcats packages. Only ggplot2 was missing to complete the tidyverse.

PS: 89 fm is not a band, it was just a slot reserved for the radio station’s advertising.

Extracting the popularity of the Lollapalooza bands on Spotify

Now let’s use the Rspotify package to extract the popularity of the bands listed in the lolla2017_programacao data.frame. For that, I used an official Spotify playlist made by the Lollapalooza team itself. This playlist is identified by the id 1mHoPn6JpbtWtoBuvSXrVm in the Spotify database.

meu_token <- spotifyOAuth(app_id, client_id, client_secret) # put your info provided by Spotify here.
lolla2017_playlist <- getPlaylistSongs("lollabr", "1mHoPn6JpbtWtoBuvSXrVm", token = meu_token) %>%
  mutate(artistInfo = map(artistId, getArtistinfo),
         artist = artist %>% tolower) %>%
  rename(track_popularity = popularity,
         track_id = id) %>%
  unnest(artistInfo) %>%
  select(artist, id, name, popularity, followers)

Some bands were left out of the playlist, so I searched for the artist name in the Spotify API itself to retrieve the corresponding id. The function that does this is searchArtist().

# retrieving info on the artists the playlist forgot ----------------------

possibly_searchArtist <- possibly(searchArtist, NA_character_)

artistas_fora_da_playlist <- lolla2017_programacao %>%
  filter(!artist %in% lolla2017_playlist$artist) %$%
  artist %>%
  data_frame(artist = .) %>%
  mutate(search_artist = map(artist, ~ .x %>% possibly_searchArtist),
         artist_info = map2(search_artist, artist, ~ {
           if(.x %>% is.na) {data.frame(search_artist = NA)} else {
           .x %>%
             mutate(name = name %>% tolower) %>%
             filter(name %in% .y) %>%
             head(1)
         }})) %>%
  select(-search_artist) %>%
  unnest(artist_info) %>%
  select(artist, id, name, popularity, followers)

Result

artist id name popularity followers
ricci 1EUMh6DZo2CfpolG75YQBL ricci 49 6116
jimmy eat world 3Ayl7mCk0nScecqOzvNp6s jimmy eat world 66 361785
89 fm NA NA NA NA
martin garrix 60d24wfXkVzDSfLS6hyCjZ martin garrix 85 2244761
illusionize 3RloA7E4XMItSP4FjMBv3L illusionize 46 30054

Putting it all together

Now let’s join the Lolla schedule with the Spotify info. The key is artist.

lolla2017 <- left_join(lolla2017_programacao,
                       lolla2017_playlist %>% bind_rows(artistas_fora_da_playlist),
                       by = "artist") %>%
  select(-id, -name) %>%
  dplyr::filter(followers %>% is.na %>% not)

Final dataset

dia palco artist hora_ini hora_fim popularity followers
sabado Palco Skol cage the elephant 2017-03-25 16:25:00 2017-03-25 17:25:00 72 745453
sabado Palco Skol metallica 2017-03-25 21:00:00 2017-03-25 23:00:00 80 3047126
sabado Palco Skol doctor pheabes 2017-03-25 12:05:00 2017-03-25 13:05:00 20 313
sabado Palco Skol rancid 2017-03-25 18:35:00 2017-03-25 19:35:00 60 220182
sabado Palco Skol suricato 2017-03-25 14:15:00 2017-03-25 15:15:00 41 52326
sabado Palco Onix the 1975 2017-03-25 17:30:00 2017-03-25 18:30:00 77 1600845
sabado Palco Onix the outs 2017-03-25 13:10:00 2017-03-25 14:10:00 28 3788
sabado Palco Onix the xx 2017-03-25 19:40:00 2017-03-25 20:55:00 76 2383923

Results

Hypothesis I

Hypothesis I: the organizers used the strategy of spreading band popularity evenly throughout the day.

One of the many logistical challenges of the event is allocating the bands to time slots across the four different stages.

I went to the event on Saturday and heard that Cage The Elephant had been one of the first bands to play. I knew how popular the band was (according to Spotify, more popular than The Strokes) and right away I found it odd that the event had them play so early.

lolla2017_grafico <- lolla2017 %>%
  mutate(hora = map2(hora_ini, hora_fim, ~ seq(.x, .y, 30*60) %>% floor_date("30 minutes"))) %>%
  unnest(hora) %>%
  group_by(dia, hora, palco) %>%
  summarise(artist = first(artist),
            n = n(),
            mean_popularity = mean(popularity))

lolla2017_grafico %>%
  ggplot(aes(x = ymd_hm(format(hora, "2017-03-26 %H%M")), y = mean_popularity, colour = palco)) +
  geom_line() +
  geom_point() +
  geom_point(data = lolla2017_grafico %>% filter(artist %in% "cage the elephant"), colour = "red", size = 2) +
  geom_text(data = lolla2017_grafico %>% filter(artist %in% "cage the elephant") %>% head(1), aes(label = artist), colour = "red", hjust = 0, vjust = -1) +
  facet_wrap(~dia) +
  labs(x = "Hora do dia", y = "Popularidade média") +
  theme(text = element_text(size = 16))

The plot above agrees with the common sense that the popular ones are left for the end, which doesn’t help confirm the hypothesis that Cage The Elefant was badly placed.

Hypothesis II

Hypothesis II: in terms of band popularity, Sunday was better than Saturday.

Talking with friends and acquaintances, I noticed that most of them were either going on Sunday or would rather go on Sunday if they had the chance. This made me wonder whether there really was a higher concentration of good bands on Sunday.

ggplot(lolla2017 %>%
         mutate(artist = artist %>% fct_reorder(popularity, .desc = TRUE))) +
  geom_bar(aes(x = artist, y = popularity, fill = dia), stat = "identity", position = "dodge") +
  theme(text = element_text(size = 16),
        axis.text.x = element_text(angle = 90, hjust = 1, vjust = 0.4))

ggplot(lolla2017) +
 geom_density(aes(fill = dia, x = popularity, colour = dia), fill = NA) +
  theme(text = element_text(size = 16),
        axis.text.x = element_text(angle = 90, hjust = 1, vjust = 0.4))

Conclusion: nothing suggests there was an imbalance. I think my circle of friends has some strange bias.

Hypothesis III

Hypothesis III: band popularity was balanced across the different stages.

When I questioned hypothesis I, I also thought about how hard it is to place the bands on the different stages. Since thousands of people would be fighting for space, it would be in the organizers’ interest to keep them as spread out as possible for several reasons: better queue flow, more comfort, less risk of accidents, among others, and a good way to do that would be to make the stages equally attractive so there wouldn’t be a big crowd in a single spot.

ggplot(lolla2017 %>%
         mutate(palco = palco %>% as.factor %>% fct_reorder(popularity, mean))) +
  geom_boxplot(aes(fill = palco, y = popularity, x = 1)) +
  theme(text = element_text(size = 16),
        axis.text.x = element_blank()) +
  labs(x = "")

The Skol stage had the least variation in popularity and usually had artists of medium to high popularity, but the AXE and Onix stages got the heavyweight artists. The Perry’s stage hosted the most lesser-known artists.

Final remarks

The topic covered here wasn’t useful, I agree, but we went through almost every step of a typical data analysis process. We did web scraping, used APIs, tidied the data, structured the information, created variables and made plots. Only the modeling part was left out. And it’s no coincidence that all the tidyverse packages were useful in this work.

The homework is to find some interesting data on the internet and apply the steps we learned here!